2024-04-26T19:43:25.932 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T19:43:25.932 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} --name rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112 --verbose --owner scheduled_rishabh@teuthology -- /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/orig.config.yaml 2024-04-26T19:43:25.990 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T19:43:26.081 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} email: null first_in_suite: false job_id: '7675112' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNZ74+Dji4nEdGTf+G1+r1tjbiHk5k11agowi/sxtvG2lee+FcSSeyKxEZzqAqMYFZWvUOCpbJ88fpB/Goplncc= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1ZHiyP88AahpnNFyE3urr2om9yXoPF99DDZUeHQuendNaWgnhW8SihPzfD76Yu9oH+rTo6biiOMauadRw2ID4= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJ4ay40E+CaHlLU+1/uxRQ/X7vqdPtpZN8K7JPya5CRaFk37mKhzqngXSxm4JZbsiU1sbOAzRaYkLobwTzTun0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/test_o_trunc.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_19:30:57 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T19:43:26.081 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-26T19:43:26.082 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-26T19:43:26.082 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T19:43:26.083 INFO:teuthology.task.internal:Checking packages... 2024-04-26T19:43:26.105 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-26T19:43:26.106 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T19:43:26.106 INFO:teuthology.packaging:ref: None 2024-04-26T19:43:26.106 INFO:teuthology.packaging:tag: None 2024-04-26T19:43:26.106 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-26T19:43:26.106 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:43:26.106 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-26T19:43:26.244 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-26T19:43:26.245 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T19:43:26.252 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T19:43:26.252 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T19:43:26.280 INFO:teuthology.task.internal:Saving configuration 2024-04-26T19:43:26.299 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T19:43:26.305 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T19:43:26.331 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 19:36:39.723675', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNZ74+Dji4nEdGTf+G1+r1tjbiHk5k11agowi/sxtvG2lee+FcSSeyKxEZzqAqMYFZWvUOCpbJ88fpB/Goplncc='} 2024-04-26T19:43:26.355 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 19:36:39.725692', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1ZHiyP88AahpnNFyE3urr2om9yXoPF99DDZUeHQuendNaWgnhW8SihPzfD76Yu9oH+rTo6biiOMauadRw2ID4='} 2024-04-26T19:43:26.376 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 19:36:39.724790', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJ4ay40E+CaHlLU+1/uxRQ/X7vqdPtpZN8K7JPya5CRaFk37mKhzqngXSxm4JZbsiU1sbOAzRaYkLobwTzTun0='} 2024-04-26T19:43:26.376 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T19:43:26.382 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T19:43:26.382 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T19:43:26.382 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T19:43:26.382 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T19:43:26.491 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f022684f160>, signals=[15]) 2024-04-26T19:43:26.492 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T19:43:26.498 INFO:teuthology.task.internal:Opening connections... 2024-04-26T19:43:26.498 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-04-26T19:43:26.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T19:43:26.589 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T19:43:26.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T19:43:26.664 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-04-26T19:43:26.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T19:43:26.752 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T19:43:26.758 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-04-26T19:43:26.776 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-04-26T19:43:26.777 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-04-26T19:43:26.834 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-04-26T19:43:26.834 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-04-26T19:43:26.834 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-04-26T19:43:26.834 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-04-26T19:43:26.834 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-04-26T19:43:26.834 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-04-26T19:43:26.834 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T19:43:26.834 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-04-26T19:43:26.835 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-04-26T19:43:26.835 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T19:43:26.835 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-04-26T19:43:26.835 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T19:43:26.835 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T19:43:26.835 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T19:43:26.835 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-04-26T19:43:26.857 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-04-26T19:43:26.875 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-04-26T19:43:26.875 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-04-26T19:43:26.932 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T19:43:26.933 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T19:43:26.933 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T19:43:26.933 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-04-26T19:43:26.957 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-04-26T19:43:26.975 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-04-26T19:43:26.975 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-04-26T19:43:27.032 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T19:43:27.033 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T19:43:27.034 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-04-26T19:43:27.057 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T19:43:27.068 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T19:43:27.080 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T19:43:27.080 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-04-26T19:43:27.082 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-04-26T19:43:27.085 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-04-26T19:43:27.101 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T19:43:27.106 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T19:43:27.106 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-04-26T19:43:27.139 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-04-26T19:43:27.141 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-04-26T19:43:27.239 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T19:43:27.317 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T19:43:27.330 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-26T19:43:27.330 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-26T19:43:27.330 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-26T19:43:27.330 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T19:43:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T19:43:27.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T19:43:27.580 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T19:43:27.580 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T19:43:27.580 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-04-26T19:43:27.597 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-437.el9.x86_64 2024-04-26T19:43:27.597 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T19:43:27.598 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T19:43:27.598 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum upgrade -y linux-firmware 2024-04-26T19:43:30.065 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-26T19:43:33.627 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 19 MB 00:02 2024-04-26T19:43:38.800 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.2 MB 00:01 2024-04-26T19:43:40.552 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-26T19:43:42.028 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-26T19:43:46.936 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-26T19:43:48.907 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-26T19:43:48.908 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-04-26T19:43:48.908 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-26T19:43:49.003 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T19:43:49.003 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T19:43:49.143 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T19:43:49.144 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T19:43:49.146 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi094.front.sepia.ceph.com to complete... 2024-04-26T19:47:01.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26 19:47:01 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T19:47:01.267 INFO:teuthology.orchestra.run.smithi094.stderr:FINISHED --2024-04-26 19:47:01-- 2024-04-26T19:47:01.267 INFO:teuthology.orchestra.run.smithi094.stderr:Total wall clock time: 3m 12s 2024-04-26T19:47:01.267 INFO:teuthology.orchestra.run.smithi094.stderr:Downloaded: 1 files, 341M in 3m 11s (1.78 MB/s) 2024-04-26T19:47:01.270 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T19:47:01.271 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T19:47:03.349 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying... ######################################## 2024-04-26T19:47:03.586 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing... ######################################## 2024-04-26T19:47:03.698 INFO:teuthology.orchestra.run.smithi094.stdout:Updating / installing... 2024-04-26T19:47:11.621 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T19:47:49.322 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T19:47:49.322 DEBUG:teuthology.orchestra.run.smithi094:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T19:47:49.394 INFO:teuthology.orchestra.run.smithi094.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T19:47:49.394 INFO:teuthology.orchestra.run.smithi094.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T19:47:49.394 INFO:teuthology.orchestra.run.smithi094.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T19:47:49.394 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T19:47:49.394 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T19:47:49.395 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T19:47:49.396 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T19:47:49.397 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T19:47:49.398 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T19:47:49.399 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T19:47:49.400 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T19:47:49.401 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T19:47:49.402 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T19:47:49.403 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T19:47:49.404 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T19:47:49.405 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T19:47:49.406 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T19:47:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T19:47:49.408 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T19:47:49.409 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T19:47:49.410 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T19:47:49.411 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T19:47:49.412 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T19:47:49.413 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T19:47:49.414 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T19:47:49.415 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T19:47:49.416 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T19:47:49.417 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T19:47:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T19:47:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T19:47:49.420 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T19:47:49.421 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T19:47:49.422 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T19:47:49.423 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T19:47:49.424 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T19:47:49.425 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T19:47:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T19:47:49.427 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T19:47:49.428 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T19:47:49.429 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T19:47:49.430 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T19:47:49.431 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T19:47:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T19:47:49.433 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T19:47:49.434 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T19:47:49.435 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T19:47:49.436 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T19:47:49.437 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T19:47:49.438 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T19:47:49.439 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T19:47:49.440 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T19:47:49.441 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T19:47:49.442 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T19:47:49.443 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T19:47:49.444 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T19:47:49.445 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T19:47:49.446 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T19:47:49.447 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T19:47:49.448 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T19:47:49.449 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T19:47:49.450 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T19:47:49.451 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T19:47:49.452 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T19:47:49.453 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T19:47:49.454 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T19:47:49.455 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T19:47:49.456 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T19:47:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T19:47:49.458 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T19:47:49.459 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T19:47:49.460 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T19:47:49.461 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T19:47:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T19:47:49.463 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T19:47:49.464 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T19:47:49.465 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T19:47:49.466 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T19:47:49.467 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T19:47:49.468 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T19:47:49.469 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T19:47:49.470 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T19:47:49.471 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T19:47:49.472 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T19:47:49.473 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T19:47:49.474 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T19:47:49.475 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T19:47:49.476 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T19:47:49.477 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T19:47:49.478 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T19:47:49.479 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T19:47:49.480 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T19:47:49.481 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T19:47:49.482 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T19:47:49.483 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T19:47:49.484 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T19:47:49.485 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T19:47:49.486 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T19:47:49.487 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T19:47:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T19:47:49.489 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T19:47:49.490 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T19:47:49.491 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T19:47:49.492 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T19:47:49.493 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T19:47:49.494 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T19:47:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T19:47:49.496 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T19:47:49.497 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T19:47:49.498 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T19:47:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T19:47:49.500 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T19:47:49.501 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T19:47:49.502 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T19:47:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T19:47:49.504 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T19:47:49.505 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T19:47:49.506 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T19:47:49.507 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T19:47:49.508 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T19:47:49.509 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T19:47:49.510 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T19:47:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T19:47:49.512 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T19:47:49.513 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T19:47:49.514 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T19:47:49.515 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T19:47:49.516 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T19:47:49.517 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T19:47:49.518 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T19:47:49.519 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T19:47:49.520 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T19:47:49.521 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T19:47:49.522 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T19:47:49.523 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T19:47:49.524 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T19:47:49.525 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T19:47:49.526 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T19:47:49.527 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T19:47:49.528 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T19:47:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T19:47:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T19:47:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T19:47:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T19:47:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T19:47:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T19:47:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T19:47:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T19:47:49.529 INFO:teuthology.orchestra.run.smithi094.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T19:47:49.533 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T19:47:49.533 DEBUG:teuthology.orchestra.run.smithi094:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T19:47:49.555 INFO:teuthology.orchestra.run.smithi094.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T19:47:49.555 INFO:teuthology.orchestra.run.smithi094.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T19:47:49.555 INFO:teuthology.orchestra.run.smithi094.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T19:47:49.555 INFO:teuthology.orchestra.run.smithi094.stdout:fi 2024-04-26T19:47:49.555 INFO:teuthology.orchestra.run.smithi094.stdout:for file in vmlinuz System.map config; do 2024-04-26T19:47:49.555 INFO:teuthology.orchestra.run.smithi094.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout: fi 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout:done 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout:fi 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout:fi 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T19:47:49.556 INFO:teuthology.orchestra.run.smithi094.stdout:fi 2024-04-26T19:47:49.557 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2024-04-26T19:47:49.627 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.06 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 68.el9 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Group : Unspecified 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 8264254 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : builder@centos.org 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2024-04-26T19:47:49.628 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T19:47:49.629 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2024-04-26T19:47:49.629 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2024-04-26T19:47:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:47:49.629 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T19:47:49.629 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T19:47:49.629 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T19:47:49.629 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2024-04-26T19:47:49.629 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:47:49.629 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T19:47:49.630 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T19:47:49.630 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T19:47:49.630 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T19:47:50.215 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2024-04-26T19:47:51.539 INFO:teuthology.orchestra.run.smithi094.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T19:47:51.565 INFO:teuthology.orchestra.run.smithi094.stderr:done 2024-04-26T19:47:51.568 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-04-26T19:47:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.WlEXrLvuRg 2024-04-26T19:47:51.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WlEXrLvuRg 2024-04-26T19:47:51.664 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.WlEXrLvuRg 2024-04-26T19:47:51.807 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.WlEXrLvuRg is 6KB 2024-04-26T19:47:51.855 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.WlEXrLvuRg 2024-04-26T19:47:51.870 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T19:47:51.932 INFO:teuthology.orchestra.run.smithi094.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T19:47:51.933 INFO:teuthology.orchestra.run.smithi094.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T19:47:51.933 INFO:teuthology.orchestra.run.smithi094.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T19:47:51.933 INFO:teuthology.orchestra.run.smithi094.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T19:47:51.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T19:47:52.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2024-04-26T19:47:52.189 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-04-26T19:48:22.111 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T19:48:22.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-26T19:48:22.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T19:48:40.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-04-26T19:48:49.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-26T19:48:49.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T19:49:49.611 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T19:50:01.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-26T19:50:01.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T19:50:01.946 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-04-26T19:50:02.634 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-26T19:50:02.634 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T19:50:02.634 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-04-26T19:50:02.685 INFO:teuthology.orchestra.run.smithi094.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T19:50:02.685 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T19:50:02.685 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T19:50:02.685 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T19:50:03.686 DEBUG:teuthology.parallel:result is None 2024-04-26T19:50:03.687 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T19:50:03.695 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T19:50:03.696 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T19:50:03.704 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T19:50:03.706 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T19:50:03.751 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T19:50:03.757 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T19:50:03.780 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T19:50:03.780 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T19:50:03.787 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T19:50:03.790 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T19:50:03.859 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T19:50:03.867 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T19:50:03.867 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T19:50:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T19:50:03.884 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T19:50:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T19:50:03.901 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T19:50:03.908 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T19:50:03.927 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T19:50:03.958 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T19:50:03.980 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T19:50:03.989 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T19:50:03.997 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T19:50:04.035 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T19:50:04.088 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T19:50:04.092 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T19:50:04.100 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T19:50:04.100 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T19:50:04.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T19:50:04.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T19:50:04.183 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T19:50:04.193 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T19:50:04.193 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T19:50:04.199 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T19:50:04.202 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T19:50:04.246 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T19:50:04.327 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T19:50:04.418 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T19:50:04.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T19:50:04.589 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T19:50:04.589 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T19:50:04.669 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T19:50:04.713 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T19:50:04.759 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T19:50:04.844 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T19:50:04.889 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T19:50:04.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T19:50:04.957 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T19:50:05.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T19:50:05.056 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T19:50:05.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T19:50:05.182 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T19:50:05.182 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T19:50:05.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-04-26T19:50:05.256 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-04-26T19:50:05.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-04-26T19:50:05.291 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T19:50:05.320 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T19:50:05.356 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T19:50:05.829 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T19:50:05.837 INFO:teuthology.task.internal:Starting timer... 2024-04-26T19:50:05.838 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T19:50:05.875 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T19:50:05.907 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T19:50:05.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-04-26T19:50:06.012 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-04-26T19:50:06.016 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-04-26T19:50:06.079 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-04-26T19:50:06.081 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-04-26T19:50:06.135 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-04-26T19:50:06.137 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T19:50:06.138 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-04-26T19:50:06.198 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-04-26T19:50:06.198 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-04-26T19:50:06.221 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-04-26T19:50:06.222 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-04-26T19:50:06.241 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-04-26T19:50:06.241 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-04-26T19:50:06.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T19:50:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T19:50:06.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T19:50:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T19:50:06.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T19:50:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T19:50:06.363 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T19:50:06.363 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-04-26T19:50:06.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-04-26T19:50:06.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-04-26T19:50:06.485 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T19:50:06.495 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T19:50:06.512 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T19:50:06.513 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi103.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T19:55:11.112 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-04-26T19:55:11.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-26T19:55:11.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T19:55:11.227 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-04-26T19:55:11.374 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-04-26T19:55:11.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T19:55:11.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T19:55:11.451 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-26T19:55:11.532 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T19:55:11.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-26T19:55:11.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T19:55:11.609 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-04-26T19:55:11.693 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-26T19:55:11.693 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T19:55:11.707 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T19:55:11.707 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T19:55:11.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T19:55:11.715 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T19:55:11.716 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T19:55:11.718 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T19:55:11.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T19:55:11.747 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T19:55:11.761 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T19:55:11.761 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T19:55:11.774 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T19:55:11.786 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-04-26T19:55:11.796 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-04-26T19:55:11.797 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-04-26T19:55:11.800 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T19:55:11.807 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-04-26T19:55:11.811 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T19:55:11.820 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T19:55:11.824 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T19:55:11.833 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T19:55:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T19:55:11.881 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-26T19:55:11.885 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T19:55:11.885 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-26T19:55:11.885 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:11.885 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:11.886 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:11.886 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:11.888 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-04-26T19:55:11.891 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T19:55:11.891 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-04-26T19:55:11.891 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:11.891 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:11.891 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:11.892 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:11.915 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-04-26T19:55:11.947 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-04-26T19:55:11.992 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T19:55:12.033 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T19:55:12.158 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-04-26T19:55:12.166 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T19:55:12.166 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-04-26T19:55:12.166 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:12.166 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:12.166 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:12.166 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T19:55:12.167 INFO:teuthology.run_tasks:Running task install... 2024-04-26T19:55:12.175 DEBUG:teuthology.task.install:project ceph 2024-04-26T19:55:12.176 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-26T19:55:12.176 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-26T19:55:12.176 INFO:teuthology.task.install:Using flavor: default 2024-04-26T19:55:12.182 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T19:55:12.182 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T19:55:12.182 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T19:55:12.182 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:55:12.184 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T19:55:12.185 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:55:12.185 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T19:55:12.186 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:55:12.409 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T19:55:12.409 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T19:55:12.413 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T19:55:12.413 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T19:55:12.416 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T19:55:12.417 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T19:55:12.677 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T19:55:12.677 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T19:55:12.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T19:55:12.682 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T19:55:12.682 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T19:55:12.682 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T19:55:12.707 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T19:55:12.707 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T19:55:12.712 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T19:55:12.712 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T19:55:12.741 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T19:55:12.741 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T19:55:12.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T19:55:12.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T19:55:12.779 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T19:55:12.831 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T19:55:12.832 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T19:55:12.850 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T19:55:12.856 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T19:55:12.914 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-04-26T19:55:12.916 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-04-26T19:55:12.921 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-04-26T19:55:12.923 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-04-26T19:55:12.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T19:55:13.074 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T19:55:13.143 INFO:teuthology.orchestra.run.smithi204.stdout:48 files removed 2024-04-26T19:55:13.151 INFO:teuthology.orchestra.run.smithi103.stdout:48 files removed 2024-04-26T19:55:13.154 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-04-26T19:55:13.159 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-04-26T19:55:13.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T19:55:13.178 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T19:55:13.480 INFO:teuthology.orchestra.run.smithi094.stdout:48 files removed 2024-04-26T19:55:13.519 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T19:55:14.099 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 126 kB/s | 84 kB 00:00 2024-04-26T19:55:14.154 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 117 kB/s | 84 kB 00:00 2024-04-26T19:55:14.558 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 137 kB/s | 84 kB 00:00 2024-04-26T19:55:14.591 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 26 kB/s | 12 kB 00:00 2024-04-26T19:55:14.638 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 26 kB/s | 12 kB 00:00 2024-04-26T19:55:14.942 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-26T19:55:15.055 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-26T19:55:15.143 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-04-26T19:55:15.550 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-26T19:55:15.768 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 3.5 kB/s | 1.9 kB 00:00 2024-04-26T19:55:15.908 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-04-26T19:55:16.684 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.0 MB 00:00 2024-04-26T19:55:17.815 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-26T19:55:17.885 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-26T19:55:21.235 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 19 MB 00:03 2024-04-26T19:55:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.2 MB 00:00 2024-04-26T19:55:22.711 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.2 MB 00:01 2024-04-26T19:55:23.808 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-26T19:55:24.354 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-26T19:55:25.535 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-26T19:55:25.749 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-26T19:55:25.953 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.2 MB 00:00 2024-04-26T19:55:28.937 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 10 kB/s | 16 kB 00:01 2024-04-26T19:55:30.406 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-26T19:55:30.527 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-26T19:55:30.838 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-26T19:55:32.619 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T19:55:32.619 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T19:55:32.630 INFO:teuthology.orchestra.run.smithi103.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T19:55:32.630 INFO:teuthology.orchestra.run.smithi103.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T19:55:32.631 INFO:teuthology.orchestra.run.smithi103.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T19:55:32.632 INFO:teuthology.orchestra.run.smithi103.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T19:55:32.633 INFO:teuthology.orchestra.run.smithi103.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T19:55:32.636 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T19:55:32.636 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T19:55:32.696 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-26T19:55:32.721 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-26T19:55:32.721 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-26T19:55:32.721 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-26T19:55:32.721 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-26T19:55:32.721 INFO:teuthology.orchestra.run.smithi103.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T19:55:32.721 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T19:55:32.721 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T19:55:32.722 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T19:55:32.723 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T19:55:32.724 INFO:teuthology.orchestra.run.smithi103.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T19:55:32.725 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T19:55:32.726 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T19:55:32.727 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T19:55:32.728 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T19:55:32.729 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T19:55:32.730 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T19:55:32.731 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T19:55:32.732 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-04-26T19:55:32.733 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T19:55:32.733 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:55:32.733 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-26T19:55:32.733 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-26T19:55:32.733 INFO:teuthology.orchestra.run.smithi103.stdout:Install 148 Packages 2024-04-26T19:55:32.733 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 4 Packages 2024-04-26T19:55:32.733 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:55:32.733 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 222 M 2024-04-26T19:55:32.734 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-26T19:55:32.948 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T19:55:32.948 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T19:55:32.959 INFO:teuthology.orchestra.run.smithi204.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T19:55:32.960 INFO:teuthology.orchestra.run.smithi204.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T19:55:32.961 INFO:teuthology.orchestra.run.smithi204.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T19:55:32.962 INFO:teuthology.orchestra.run.smithi204.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T19:55:32.963 INFO:teuthology.orchestra.run.smithi204.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T19:55:32.965 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T19:55:32.966 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T19:55:33.027 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-04-26T19:55:33.044 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-04-26T19:55:33.044 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-04-26T19:55:33.044 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-04-26T19:55:33.044 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-04-26T19:55:33.044 INFO:teuthology.orchestra.run.smithi204.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T19:55:33.045 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T19:55:33.046 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-04-26T19:55:33.047 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T19:55:33.048 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T19:55:33.049 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T19:55:33.050 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T19:55:33.051 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T19:55:33.052 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T19:55:33.053 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T19:55:33.054 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-04-26T19:55:33.055 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T19:55:33.056 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:55:33.056 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-04-26T19:55:33.056 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-04-26T19:55:33.056 INFO:teuthology.orchestra.run.smithi204.stdout:Install 148 Packages 2024-04-26T19:55:33.056 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 4 Packages 2024-04-26T19:55:33.056 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:55:33.056 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 222 M 2024-04-26T19:55:33.056 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-04-26T19:55:33.569 INFO:teuthology.orchestra.run.smithi103.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-26T19:55:33.874 INFO:teuthology.orchestra.run.smithi204.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-04-26T19:55:34.328 INFO:teuthology.orchestra.run.smithi103.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.1 MB/s | 889 kB 00:00 2024-04-26T19:55:35.103 INFO:teuthology.orchestra.run.smithi103.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 187 kB/s | 145 kB 00:00 2024-04-26T19:55:35.329 INFO:teuthology.orchestra.run.smithi103.stdout:(4/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 2.5 MB/s | 5.4 MB 00:02 2024-04-26T19:55:35.399 INFO:teuthology.orchestra.run.smithi204.stdout:(2/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 2.8 MB/s | 5.4 MB 00:01 2024-04-26T19:55:35.507 INFO:teuthology.orchestra.run.smithi204.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 1.3 MB/s | 145 kB 00:00 2024-04-26T19:55:35.762 INFO:teuthology.orchestra.run.smithi103.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 3.5 MB/s | 1.5 MB 00:00 2024-04-26T19:55:36.229 INFO:teuthology.orchestra.run.smithi103.stdout:(6/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 2.1 MB/s | 2.4 MB 00:01 2024-04-26T19:55:36.357 INFO:teuthology.orchestra.run.smithi204.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 2.8 MB/s | 2.4 MB 00:00 2024-04-26T19:55:36.381 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-26T19:55:36.385 INFO:teuthology.orchestra.run.smithi204.stdout:(5/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 354 kB/s | 889 kB 00:02 2024-04-26T19:55:36.893 INFO:teuthology.orchestra.run.smithi204.stdout:(6/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-04-26T19:55:36.938 INFO:teuthology.orchestra.run.smithi103.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 4.0 MB/s | 4.7 MB 00:01 2024-04-26T19:55:38.763 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T19:55:38.764 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T19:55:38.784 INFO:teuthology.orchestra.run.smithi094.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T19:55:38.786 INFO:teuthology.orchestra.run.smithi094.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T19:55:38.787 INFO:teuthology.orchestra.run.smithi094.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T19:55:38.788 INFO:teuthology.orchestra.run.smithi094.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T19:55:38.791 INFO:teuthology.orchestra.run.smithi094.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T19:55:38.794 INFO:teuthology.orchestra.run.smithi094.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T19:55:38.795 INFO:teuthology.orchestra.run.smithi094.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T19:55:38.865 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-04-26T19:55:38.885 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-04-26T19:55:38.885 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-04-26T19:55:38.885 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-04-26T19:55:38.885 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-04-26T19:55:38.885 INFO:teuthology.orchestra.run.smithi094.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T19:55:38.885 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T19:55:38.885 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T19:55:38.885 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T19:55:38.885 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T19:55:38.886 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T19:55:38.887 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T19:55:38.888 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T19:55:38.888 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T19:55:38.888 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T19:55:38.888 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T19:55:38.889 INFO:teuthology.orchestra.run.smithi094.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T19:55:38.890 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T19:55:38.891 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T19:55:38.892 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T19:55:38.893 INFO:teuthology.orchestra.run.smithi094.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T19:55:38.894 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T19:55:38.895 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T19:55:38.896 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T19:55:38.897 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout:Install 148 Packages 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 4 Packages 2024-04-26T19:55:38.898 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:55:38.899 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 222 M 2024-04-26T19:55:38.899 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-04-26T19:55:39.604 INFO:teuthology.orchestra.run.smithi103.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 4.0 MB/s | 11 MB 00:02 2024-04-26T19:55:39.713 INFO:teuthology.orchestra.run.smithi103.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 233 kB/s | 25 kB 00:00 2024-04-26T19:55:40.158 INFO:teuthology.orchestra.run.smithi094.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-26T19:55:41.033 INFO:teuthology.orchestra.run.smithi094.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.0 MB/s | 889 kB 00:00 2024-04-26T19:55:41.217 INFO:teuthology.orchestra.run.smithi094.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 794 kB/s | 145 kB 00:00 2024-04-26T19:55:42.950 INFO:teuthology.orchestra.run.smithi094.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 1.4 MB/s | 2.4 MB 00:01 2024-04-26T19:55:44.097 INFO:teuthology.orchestra.run.smithi103.stdout:(10/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 2.1 MB/s | 17 MB 00:07 2024-04-26T19:55:44.226 INFO:teuthology.orchestra.run.smithi204.stdout:(7/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x86 2.3 MB/s | 17 MB 00:07 2024-04-26T19:55:44.379 INFO:teuthology.orchestra.run.smithi103.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 795 kB/s | 225 kB 00:00 2024-04-26T19:55:44.409 INFO:teuthology.orchestra.run.smithi094.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 1.0 MB/s | 1.5 MB 00:01 2024-04-26T19:55:44.488 INFO:teuthology.orchestra.run.smithi103.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 249 kB/s | 27 kB 00:00 2024-04-26T19:55:44.597 INFO:teuthology.orchestra.run.smithi103.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 293 kB/s | 32 kB 00:00 2024-04-26T19:55:45.310 INFO:teuthology.orchestra.run.smithi204.stdout:(8/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 539 kB/s | 4.7 MB 00:08 2024-04-26T19:55:45.322 INFO:teuthology.orchestra.run.smithi103.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 1.0 MB/s | 747 kB 00:00 2024-04-26T19:55:45.393 INFO:teuthology.orchestra.run.smithi204.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 303 kB/s | 25 kB 00:00 2024-04-26T19:55:45.505 INFO:teuthology.orchestra.run.smithi103.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 891 kB/s | 163 kB 00:00 2024-04-26T19:55:45.697 INFO:teuthology.orchestra.run.smithi103.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 663 kB/s | 127 kB 00:00 2024-04-26T19:55:46.672 INFO:teuthology.orchestra.run.smithi103.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 514 kB/s | 501 kB 00:00 2024-04-26T19:55:46.976 INFO:teuthology.orchestra.run.smithi094.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 758 kB/s | 5.4 MB 00:07 2024-04-26T19:55:47.365 INFO:teuthology.orchestra.run.smithi103.stdout:(18/152): ceph-common-19.0.0-3306.gbc612ffb.el9 1.5 MB/s | 21 MB 00:14 2024-04-26T19:55:47.965 INFO:teuthology.orchestra.run.smithi103.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 75 kB/s | 45 kB 00:00 2024-04-26T19:55:48.424 INFO:teuthology.orchestra.run.smithi103.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 295 kB/s | 135 kB 00:00 2024-04-26T19:55:48.849 INFO:teuthology.orchestra.run.smithi103.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 380 kB/s | 161 kB 00:00 2024-04-26T19:55:49.600 INFO:teuthology.orchestra.run.smithi103.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 428 kB/s | 321 kB 00:00 2024-04-26T19:55:49.701 INFO:teuthology.orchestra.run.smithi094.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 910 kB/s | 4.7 MB 00:05 2024-04-26T19:55:50.250 INFO:teuthology.orchestra.run.smithi103.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 458 kB/s | 298 kB 00:00 2024-04-26T19:55:50.550 INFO:teuthology.orchestra.run.smithi103.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 331 kB/s | 99 kB 00:00 2024-04-26T19:55:50.759 INFO:teuthology.orchestra.run.smithi103.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 409 kB/s | 85 kB 00:00 2024-04-26T19:55:54.368 INFO:teuthology.orchestra.run.smithi204.stdout:(10/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 1.0 MB/s | 11 MB 00:10 2024-04-26T19:55:54.651 INFO:teuthology.orchestra.run.smithi204.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 795 kB/s | 225 kB 00:00 2024-04-26T19:55:54.777 INFO:teuthology.orchestra.run.smithi204.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 216 kB/s | 27 kB 00:00 2024-04-26T19:55:55.192 INFO:teuthology.orchestra.run.smithi103.stdout:(26/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 714 kB/s | 3.1 MB 00:04 2024-04-26T19:55:55.277 INFO:teuthology.orchestra.run.smithi204.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 63 kB/s | 32 kB 00:00 2024-04-26T19:55:55.509 INFO:teuthology.orchestra.run.smithi103.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 539 kB/s | 171 kB 00:00 2024-04-26T19:55:55.626 INFO:teuthology.orchestra.run.smithi103.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 293 kB/s | 34 kB 00:00 2024-04-26T19:55:55.926 INFO:teuthology.orchestra.run.smithi103.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 474 kB/s | 142 kB 00:00 2024-04-26T19:55:56.677 INFO:teuthology.orchestra.run.smithi204.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 533 kB/s | 747 kB 00:01 2024-04-26T19:55:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:(30/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 545 kB/s | 5.4 MB 00:10 2024-04-26T19:55:57.444 INFO:teuthology.orchestra.run.smithi204.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 213 kB/s | 163 kB 00:00 2024-04-26T19:55:57.686 INFO:teuthology.orchestra.run.smithi204.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 525 kB/s | 127 kB 00:00 2024-04-26T19:55:58.419 INFO:teuthology.orchestra.run.smithi204.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 683 kB/s | 501 kB 00:00 2024-04-26T19:55:58.834 INFO:teuthology.orchestra.run.smithi094.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 1.2 MB/s | 11 MB 00:09 2024-04-26T19:55:58.909 INFO:teuthology.orchestra.run.smithi094.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 340 kB/s | 25 kB 00:00 2024-04-26T19:55:59.876 INFO:teuthology.orchestra.run.smithi103.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 917 kB/s | 3.5 MB 00:03 2024-04-26T19:56:00.160 INFO:teuthology.orchestra.run.smithi103.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 878 kB/s | 249 kB 00:00 2024-04-26T19:56:00.269 INFO:teuthology.orchestra.run.smithi103.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 456 kB/s | 49 kB 00:00 2024-04-26T19:56:00.421 INFO:teuthology.orchestra.run.smithi103.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 108 kB/s | 16 kB 00:00 2024-04-26T19:56:00.727 INFO:teuthology.orchestra.run.smithi103.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 891 kB/s | 273 kB 00:00 2024-04-26T19:56:01.719 INFO:teuthology.orchestra.run.smithi103.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 769 kB/s | 762 kB 00:00 2024-04-26T19:56:01.811 INFO:teuthology.orchestra.run.smithi204.stdout:(18/152): ceph-common-19.0.0-3306.gbc612ffb.el9 764 kB/s | 21 MB 00:28 2024-04-26T19:56:01.820 INFO:teuthology.orchestra.run.smithi103.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 182 kB/s | 18 kB 00:00 2024-04-26T19:56:01.878 INFO:teuthology.orchestra.run.smithi103.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 80 kB 00:00 2024-04-26T19:56:01.979 INFO:teuthology.orchestra.run.smithi103.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.8 MB/s | 803 kB 00:00 2024-04-26T19:56:01.986 INFO:teuthology.orchestra.run.smithi204.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 258 kB/s | 45 kB 00:00 2024-04-26T19:56:02.037 INFO:teuthology.orchestra.run.smithi103.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2024-04-26T19:56:02.112 INFO:teuthology.orchestra.run.smithi103.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-26T19:56:02.246 INFO:teuthology.orchestra.run.smithi103.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-04-26T19:56:02.278 INFO:teuthology.orchestra.run.smithi204.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 463 kB/s | 135 kB 00:00 2024-04-26T19:56:02.321 INFO:teuthology.orchestra.run.smithi103.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-26T19:56:02.388 INFO:teuthology.orchestra.run.smithi103.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-26T19:56:02.456 INFO:teuthology.orchestra.run.smithi103.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-26T19:56:02.520 INFO:teuthology.orchestra.run.smithi204.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 669 kB/s | 161 kB 00:00 2024-04-26T19:56:02.539 INFO:teuthology.orchestra.run.smithi103.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-04-26T19:56:02.987 INFO:teuthology.orchestra.run.smithi204.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 688 kB/s | 321 kB 00:00 2024-04-26T19:56:03.081 INFO:teuthology.orchestra.run.smithi103.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 1.2 MB/s | 689 kB 00:00 2024-04-26T19:56:03.315 INFO:teuthology.orchestra.run.smithi103.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 943 kB 00:00 2024-04-26T19:56:03.373 INFO:teuthology.orchestra.run.smithi103.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-26T19:56:03.387 INFO:teuthology.orchestra.run.smithi204.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 744 kB/s | 298 kB 00:00 2024-04-26T19:56:03.432 INFO:teuthology.orchestra.run.smithi103.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 369 kB/s | 21 kB 00:00 2024-04-26T19:56:03.491 INFO:teuthology.orchestra.run.smithi103.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-04-26T19:56:03.518 INFO:teuthology.orchestra.run.smithi094.stdout:(10/152): ceph-common-19.0.0-3306.gbc612ffb.el9 910 kB/s | 21 MB 00:23 2024-04-26T19:56:03.541 INFO:teuthology.orchestra.run.smithi103.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2024-04-26T19:56:03.546 INFO:teuthology.orchestra.run.smithi204.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 628 kB/s | 99 kB 00:00 2024-04-26T19:56:03.833 INFO:teuthology.orchestra.run.smithi103.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 10 MB/s | 3.0 MB 00:00 2024-04-26T19:56:03.883 INFO:teuthology.orchestra.run.smithi103.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-26T19:56:03.888 INFO:teuthology.orchestra.run.smithi204.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 250 kB/s | 85 kB 00:00 2024-04-26T19:56:03.942 INFO:teuthology.orchestra.run.smithi103.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-04-26T19:56:04.018 INFO:teuthology.orchestra.run.smithi094.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 451 kB/s | 225 kB 00:00 2024-04-26T19:56:04.067 INFO:teuthology.orchestra.run.smithi103.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-26T19:56:04.118 INFO:teuthology.orchestra.run.smithi103.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 909 kB/s | 45 kB 00:00 2024-04-26T19:56:04.143 INFO:teuthology.orchestra.run.smithi094.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 217 kB/s | 27 kB 00:00 2024-04-26T19:56:04.218 INFO:teuthology.orchestra.run.smithi103.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-04-26T19:56:04.252 INFO:teuthology.orchestra.run.smithi094.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 295 kB/s | 32 kB 00:00 2024-04-26T19:56:04.302 INFO:teuthology.orchestra.run.smithi103.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 582 kB 00:00 2024-04-26T19:56:04.343 INFO:teuthology.orchestra.run.smithi103.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 614 kB/s | 25 kB 00:00 2024-04-26T19:56:04.386 INFO:teuthology.orchestra.run.smithi103.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-26T19:56:04.444 INFO:teuthology.orchestra.run.smithi103.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-26T19:56:04.503 INFO:teuthology.orchestra.run.smithi103.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-26T19:56:04.553 INFO:teuthology.orchestra.run.smithi103.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 706 kB/s | 35 kB 00:00 2024-04-26T19:56:05.054 INFO:teuthology.orchestra.run.smithi103.stdout:(65/152): openblas-openmp-0.3.21-2.el9.x86_64.r 9.5 MB/s | 4.7 MB 00:00 2024-04-26T19:56:05.388 INFO:teuthology.orchestra.run.smithi094.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 658 kB/s | 747 kB 00:01 2024-04-26T19:56:05.437 INFO:teuthology.orchestra.run.smithi103.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-26T19:56:05.496 INFO:teuthology.orchestra.run.smithi204.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 777 kB/s | 5.4 MB 00:07 2024-04-26T19:56:05.596 INFO:teuthology.orchestra.run.smithi103.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-26T19:56:05.646 INFO:teuthology.orchestra.run.smithi094.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 634 kB/s | 163 kB 00:00 2024-04-26T19:56:05.679 INFO:teuthology.orchestra.run.smithi103.stdout:(68/152): ceph-mgr-diskprediction-local-19.0.0- 851 kB/s | 7.4 MB 00:08 2024-04-26T19:56:05.721 INFO:teuthology.orchestra.run.smithi103.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 7.1 MB/s | 911 kB 00:00 2024-04-26T19:56:05.830 INFO:teuthology.orchestra.run.smithi204.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 512 kB/s | 171 kB 00:00 2024-04-26T19:56:05.863 INFO:teuthology.orchestra.run.smithi094.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 588 kB/s | 127 kB 00:00 2024-04-26T19:56:05.864 INFO:teuthology.orchestra.run.smithi103.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 726 kB/s | 133 kB 00:00 2024-04-26T19:56:05.947 INFO:teuthology.orchestra.run.smithi204.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 293 kB/s | 34 kB 00:00 2024-04-26T19:56:06.039 INFO:teuthology.orchestra.run.smithi103.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-04-26T19:56:06.098 INFO:teuthology.orchestra.run.smithi103.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T19:56:06.197 INFO:teuthology.orchestra.run.smithi103.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-04-26T19:56:06.230 INFO:teuthology.orchestra.run.smithi204.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 501 kB/s | 142 kB 00:00 2024-04-26T19:56:06.256 INFO:teuthology.orchestra.run.smithi103.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-26T19:56:06.315 INFO:teuthology.orchestra.run.smithi103.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-26T19:56:06.365 INFO:teuthology.orchestra.run.smithi103.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 512 kB/s | 26 kB 00:00 2024-04-26T19:56:06.415 INFO:teuthology.orchestra.run.smithi103.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-04-26T19:56:06.474 INFO:teuthology.orchestra.run.smithi103.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-26T19:56:06.524 INFO:teuthology.orchestra.run.smithi103.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-04-26T19:56:06.580 INFO:teuthology.orchestra.run.smithi094.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 699 kB/s | 501 kB 00:00 2024-04-26T19:56:06.908 INFO:teuthology.orchestra.run.smithi103.stdout:(80/152): python3-babel-2.9.1-2.el9.noarch.rpm 7.4 MB/s | 6.0 MB 00:00 2024-04-26T19:56:07.025 INFO:teuthology.orchestra.run.smithi103.stdout:(81/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-04-26T19:56:07.084 INFO:teuthology.orchestra.run.smithi103.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-26T19:56:07.109 INFO:teuthology.orchestra.run.smithi103.stdout:(83/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 250 kB 00:00 2024-04-26T19:56:07.134 INFO:teuthology.orchestra.run.smithi103.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T19:56:07.159 INFO:teuthology.orchestra.run.smithi103.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-04-26T19:56:07.193 INFO:teuthology.orchestra.run.smithi103.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-26T19:56:07.243 INFO:teuthology.orchestra.run.smithi103.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-04-26T19:56:07.268 INFO:teuthology.orchestra.run.smithi103.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-26T19:56:07.285 INFO:teuthology.orchestra.run.smithi103.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-26T19:56:07.311 INFO:teuthology.orchestra.run.smithi103.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 54 kB 00:00 2024-04-26T19:56:07.353 INFO:teuthology.orchestra.run.smithi103.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-26T19:56:07.403 INFO:teuthology.orchestra.run.smithi103.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 886 kB/s | 44 kB 00:00 2024-04-26T19:56:07.462 INFO:teuthology.orchestra.run.smithi103.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-26T19:56:07.512 INFO:teuthology.orchestra.run.smithi103.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-04-26T19:56:07.563 INFO:teuthology.orchestra.run.smithi103.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 951 kB/s | 47 kB 00:00 2024-04-26T19:56:07.621 INFO:teuthology.orchestra.run.smithi103.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-26T19:56:07.672 INFO:teuthology.orchestra.run.smithi103.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-26T19:56:07.722 INFO:teuthology.orchestra.run.smithi103.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-26T19:56:07.772 INFO:teuthology.orchestra.run.smithi103.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-04-26T19:56:07.823 INFO:teuthology.orchestra.run.smithi103.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2024-04-26T19:56:07.873 INFO:teuthology.orchestra.run.smithi103.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-04-26T19:56:07.965 INFO:teuthology.orchestra.run.smithi103.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.2 MB/s | 863 kB 00:00 2024-04-26T19:56:08.497 INFO:teuthology.orchestra.run.smithi094.stdout:(18/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 801 kB/s | 17 MB 00:21 2024-04-26T19:56:08.557 INFO:teuthology.orchestra.run.smithi103.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 926 kB/s | 548 kB 00:00 2024-04-26T19:56:08.731 INFO:teuthology.orchestra.run.smithi094.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 194 kB/s | 45 kB 00:00 2024-04-26T19:56:08.939 INFO:teuthology.orchestra.run.smithi094.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 652 kB/s | 135 kB 00:00 2024-04-26T19:56:09.065 INFO:teuthology.orchestra.run.smithi103.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 608 kB/s | 309 kB 00:00 2024-04-26T19:56:09.223 INFO:teuthology.orchestra.run.smithi094.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 570 kB/s | 161 kB 00:00 2024-04-26T19:56:09.341 INFO:teuthology.orchestra.run.smithi103.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-04-26T19:56:09.715 INFO:teuthology.orchestra.run.smithi094.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 654 kB/s | 321 kB 00:00 2024-04-26T19:56:09.781 INFO:teuthology.orchestra.run.smithi204.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 537 kB/s | 3.1 MB 00:05 2024-04-26T19:56:09.973 INFO:teuthology.orchestra.run.smithi204.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 969 kB/s | 3.5 MB 00:03 2024-04-26T19:56:10.107 INFO:teuthology.orchestra.run.smithi094.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 762 kB/s | 298 kB 00:00 2024-04-26T19:56:10.273 INFO:teuthology.orchestra.run.smithi204.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 830 kB/s | 249 kB 00:00 2024-04-26T19:56:10.324 INFO:teuthology.orchestra.run.smithi094.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 460 kB/s | 99 kB 00:00 2024-04-26T19:56:10.398 INFO:teuthology.orchestra.run.smithi204.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 395 kB/s | 49 kB 00:00 2024-04-26T19:56:10.515 INFO:teuthology.orchestra.run.smithi204.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 141 kB/s | 16 kB 00:00 2024-04-26T19:56:10.541 INFO:teuthology.orchestra.run.smithi094.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 395 kB/s | 85 kB 00:00 2024-04-26T19:56:10.816 INFO:teuthology.orchestra.run.smithi204.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 910 kB/s | 273 kB 00:00 2024-04-26T19:56:10.899 INFO:teuthology.orchestra.run.smithi103.stdout:(106/152): libarrow-9.0.0-11.el9.x86_64.rpm 2.8 MB/s | 4.4 MB 00:01 2024-04-26T19:56:11.166 INFO:teuthology.orchestra.run.smithi103.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-04-26T19:56:11.507 INFO:teuthology.orchestra.run.smithi204.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 1.1 MB/s | 762 kB 00:00 2024-04-26T19:56:11.508 INFO:teuthology.orchestra.run.smithi103.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-04-26T19:56:11.608 INFO:teuthology.orchestra.run.smithi204.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 182 kB/s | 18 kB 00:00 2024-04-26T19:56:11.766 INFO:teuthology.orchestra.run.smithi204.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 508 kB/s | 80 kB 00:00 2024-04-26T19:56:11.850 INFO:teuthology.orchestra.run.smithi103.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-04-26T19:56:11.941 INFO:teuthology.orchestra.run.smithi204.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-04-26T19:56:12.016 INFO:teuthology.orchestra.run.smithi204.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-04-26T19:56:12.108 INFO:teuthology.orchestra.run.smithi204.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-04-26T19:56:12.134 INFO:teuthology.orchestra.run.smithi103.stdout:(110/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.4 MB/s | 16 MB 00:04 2024-04-26T19:56:12.359 INFO:teuthology.orchestra.run.smithi103.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 297 kB/s | 151 kB 00:00 2024-04-26T19:56:12.484 INFO:teuthology.orchestra.run.smithi204.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 3.3 MB/s | 1.2 MB 00:00 2024-04-26T19:56:12.534 INFO:teuthology.orchestra.run.smithi204.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-26T19:56:12.584 INFO:teuthology.orchestra.run.smithi204.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T19:56:12.635 INFO:teuthology.orchestra.run.smithi204.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-26T19:56:12.693 INFO:teuthology.orchestra.run.smithi204.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-26T19:56:12.810 INFO:teuthology.orchestra.run.smithi103.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.2 MB/s | 839 kB 00:00 2024-04-26T19:56:12.835 INFO:teuthology.orchestra.run.smithi204.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 4.8 MB/s | 689 kB 00:00 2024-04-26T19:56:12.951 INFO:teuthology.orchestra.run.smithi103.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 927 kB/s | 548 kB 00:00 2024-04-26T19:56:13.011 INFO:teuthology.orchestra.run.smithi204.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 5.3 MB/s | 943 kB 00:00 2024-04-26T19:56:13.061 INFO:teuthology.orchestra.run.smithi204.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-26T19:56:13.103 INFO:teuthology.orchestra.run.smithi204.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 515 kB/s | 21 kB 00:00 2024-04-26T19:56:13.160 INFO:teuthology.orchestra.run.smithi103.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 124 kB/s | 43 kB 00:00 2024-04-26T19:56:13.203 INFO:teuthology.orchestra.run.smithi204.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 3.1 MB/s | 313 kB 00:00 2024-04-26T19:56:13.218 INFO:teuthology.orchestra.run.smithi103.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 121 kB/s | 32 kB 00:00 2024-04-26T19:56:13.262 INFO:teuthology.orchestra.run.smithi204.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-04-26T19:56:13.419 INFO:teuthology.orchestra.run.smithi103.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 58 kB/s | 15 kB 00:00 2024-04-26T19:56:13.442 INFO:teuthology.orchestra.run.smithi094.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 801 kB/s | 5.4 MB 00:06 2024-04-26T19:56:13.644 INFO:teuthology.orchestra.run.smithi103.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 404 kB/s | 172 kB 00:00 2024-04-26T19:56:13.709 INFO:teuthology.orchestra.run.smithi094.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 641 kB/s | 171 kB 00:00 2024-04-26T19:56:13.745 INFO:teuthology.orchestra.run.smithi204.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.2 MB/s | 3.0 MB 00:00 2024-04-26T19:56:13.787 INFO:teuthology.orchestra.run.smithi204.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 437 kB/s | 18 kB 00:00 2024-04-26T19:56:13.825 INFO:teuthology.orchestra.run.smithi094.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 294 kB/s | 34 kB 00:00 2024-04-26T19:56:13.838 INFO:teuthology.orchestra.run.smithi204.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-26T19:56:13.897 INFO:teuthology.orchestra.run.smithi204.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-26T19:56:13.947 INFO:teuthology.orchestra.run.smithi204.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 906 kB/s | 45 kB 00:00 2024-04-26T19:56:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:(29/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 928 kB/s | 3.1 MB 00:03 2024-04-26T19:56:13.978 INFO:teuthology.orchestra.run.smithi103.stdout:(118/152): ceph-test-19.0.0-3306.gbc612ffb.el9. 1.4 MB/s | 49 MB 00:34 2024-04-26T19:56:14.059 INFO:teuthology.orchestra.run.smithi094.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 610 kB/s | 142 kB 00:00 2024-04-26T19:56:14.064 INFO:teuthology.orchestra.run.smithi204.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-26T19:56:14.086 INFO:teuthology.orchestra.run.smithi103.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 537 kB/s | 358 kB 00:00 2024-04-26T19:56:14.181 INFO:teuthology.orchestra.run.smithi204.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-04-26T19:56:14.231 INFO:teuthology.orchestra.run.smithi204.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 510 kB/s | 25 kB 00:00 2024-04-26T19:56:14.245 INFO:teuthology.orchestra.run.smithi103.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 132 kB/s | 35 kB 00:00 2024-04-26T19:56:14.282 INFO:teuthology.orchestra.run.smithi204.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-26T19:56:14.303 INFO:teuthology.orchestra.run.smithi103.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 460 kB/s | 303 kB 00:00 2024-04-26T19:56:14.365 INFO:teuthology.orchestra.run.smithi204.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-26T19:56:14.424 INFO:teuthology.orchestra.run.smithi204.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-26T19:56:14.474 INFO:teuthology.orchestra.run.smithi204.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 706 kB/s | 35 kB 00:00 2024-04-26T19:56:14.620 INFO:teuthology.orchestra.run.smithi103.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 385 kB/s | 206 kB 00:00 2024-04-26T19:56:14.729 INFO:teuthology.orchestra.run.smithi103.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 340 kB/s | 144 kB 00:00 2024-04-26T19:56:14.805 INFO:teuthology.orchestra.run.smithi103.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-04-26T19:56:15.005 INFO:teuthology.orchestra.run.smithi103.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 64 kB/s | 18 kB 00:00 2024-04-26T19:56:15.016 INFO:teuthology.orchestra.run.smithi204.stdout:(65/152): openblas-openmp-0.3.21-2.el9.x86_64.r 8.7 MB/s | 4.7 MB 00:00 2024-04-26T19:56:15.072 INFO:teuthology.orchestra.run.smithi103.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 87 kB/s | 23 kB 00:00 2024-04-26T19:56:15.205 INFO:teuthology.orchestra.run.smithi103.stdout:(127/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.1 MB/s | 2.0 MB 00:00 2024-04-26T19:56:15.272 INFO:teuthology.orchestra.run.smithi103.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 73 kB/s | 19 kB 00:00 2024-04-26T19:56:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 74 kB/s | 20 kB 00:00 2024-04-26T19:56:15.425 INFO:teuthology.orchestra.run.smithi204.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-26T19:56:15.583 INFO:teuthology.orchestra.run.smithi204.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-26T19:56:15.623 INFO:teuthology.orchestra.run.smithi103.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 132 kB/s | 46 kB 00:00 2024-04-26T19:56:15.684 INFO:teuthology.orchestra.run.smithi204.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-04-26T19:56:15.742 INFO:teuthology.orchestra.run.smithi204.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-26T19:56:15.765 INFO:teuthology.orchestra.run.smithi103.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 185 kB/s | 79 kB 00:00 2024-04-26T19:56:15.890 INFO:teuthology.orchestra.run.smithi103.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-04-26T19:56:15.965 INFO:teuthology.orchestra.run.smithi103.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 169 kB/s | 58 kB 00:00 2024-04-26T19:56:16.143 INFO:teuthology.orchestra.run.smithi204.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 14 MB/s | 5.8 MB 00:00 2024-04-26T19:56:16.158 INFO:teuthology.orchestra.run.smithi103.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 61 kB/s | 16 kB 00:00 2024-04-26T19:56:16.243 INFO:teuthology.orchestra.run.smithi204.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-26T19:56:16.274 INFO:teuthology.orchestra.run.smithi103.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 535 kB/s | 272 kB 00:00 2024-04-26T19:56:16.302 INFO:teuthology.orchestra.run.smithi204.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T19:56:16.308 INFO:teuthology.orchestra.run.smithi103.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 263 kB/s | 90 kB 00:00 2024-04-26T19:56:16.425 INFO:teuthology.orchestra.run.smithi103.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 106 kB/s | 28 kB 00:00 2024-04-26T19:56:16.542 INFO:teuthology.orchestra.run.smithi103.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 115 kB/s | 31 kB 00:00 2024-04-26T19:56:16.734 INFO:teuthology.orchestra.run.smithi103.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 443 kB/s | 188 kB 00:00 2024-04-26T19:56:16.767 INFO:teuthology.orchestra.run.smithi103.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-04-26T19:56:16.802 INFO:teuthology.orchestra.run.smithi204.stdout:(73/152): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-04-26T19:56:16.809 INFO:teuthology.orchestra.run.smithi103.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 134 kB/s | 36 kB 00:00 2024-04-26T19:56:17.027 INFO:teuthology.orchestra.run.smithi204.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-04-26T19:56:17.085 INFO:teuthology.orchestra.run.smithi103.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 145 kB/s | 51 kB 00:00 2024-04-26T19:56:17.111 INFO:teuthology.orchestra.run.smithi204.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-26T19:56:17.160 INFO:teuthology.orchestra.run.smithi103.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 256 kB/s | 90 kB 00:00 2024-04-26T19:56:17.161 INFO:teuthology.orchestra.run.smithi204.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 511 kB/s | 26 kB 00:00 2024-04-26T19:56:17.220 INFO:teuthology.orchestra.run.smithi204.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 329 kB/s | 19 kB 00:00 2024-04-26T19:56:17.279 INFO:teuthology.orchestra.run.smithi204.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-26T19:56:17.284 INFO:teuthology.orchestra.run.smithi103.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 445 kB/s | 230 kB 00:00 2024-04-26T19:56:17.329 INFO:teuthology.orchestra.run.smithi204.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 697 kB/s | 35 kB 00:00 2024-04-26T19:56:17.427 INFO:teuthology.orchestra.run.smithi103.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 75 kB/s | 20 kB 00:00 2024-04-26T19:56:17.678 INFO:teuthology.orchestra.run.smithi103.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 721 kB/s | 427 kB 00:00 2024-04-26T19:56:17.711 INFO:teuthology.orchestra.run.smithi103.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 449 kB/s | 192 kB 00:00 2024-04-26T19:56:17.771 INFO:teuthology.orchestra.run.smithi204.stdout:(80/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-26T19:56:17.829 INFO:teuthology.orchestra.run.smithi204.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-26T19:56:17.897 INFO:teuthology.orchestra.run.smithi204.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.2 MB/s | 222 kB 00:00 2024-04-26T19:56:17.947 INFO:teuthology.orchestra.run.smithi204.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T19:56:17.997 INFO:teuthology.orchestra.run.smithi204.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 846 kB/s | 42 kB 00:00 2024-04-26T19:56:18.056 INFO:teuthology.orchestra.run.smithi204.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-26T19:56:18.115 INFO:teuthology.orchestra.run.smithi204.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-26T19:56:18.173 INFO:teuthology.orchestra.run.smithi204.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-26T19:56:18.215 INFO:teuthology.orchestra.run.smithi204.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-26T19:56:18.253 INFO:teuthology.orchestra.run.smithi103.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-04-26T19:56:18.257 INFO:teuthology.orchestra.run.smithi204.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-04-26T19:56:18.378 INFO:teuthology.orchestra.run.smithi103.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 11 MB/s | 1.3 MB 00:00 2024-04-26T19:56:18.545 INFO:teuthology.orchestra.run.smithi103.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-04-26T19:56:19.216 INFO:teuthology.orchestra.run.smithi204.stdout:(90/152): ceph-mgr-diskprediction-local-19.0.0- 804 kB/s | 7.4 MB 00:09 2024-04-26T19:56:19.301 INFO:teuthology.orchestra.run.smithi094.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 677 kB/s | 3.5 MB 00:05 2024-04-26T19:56:19.516 INFO:teuthology.orchestra.run.smithi204.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 139 kB/s | 42 kB 00:00 2024-04-26T19:56:19.641 INFO:teuthology.orchestra.run.smithi204.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 354 kB/s | 44 kB 00:00 2024-04-26T19:56:19.793 INFO:teuthology.orchestra.run.smithi094.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 507 kB/s | 249 kB 00:00 2024-04-26T19:56:19.800 INFO:teuthology.orchestra.run.smithi204.stdout:(93/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-04-26T19:56:19.825 INFO:teuthology.orchestra.run.smithi204.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-04-26T19:56:19.850 INFO:teuthology.orchestra.run.smithi204.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 719 kB/s | 36 kB 00:00 2024-04-26T19:56:19.909 INFO:teuthology.orchestra.run.smithi204.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-04-26T19:56:19.910 INFO:teuthology.orchestra.run.smithi094.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 425 kB/s | 49 kB 00:00 2024-04-26T19:56:19.934 INFO:teuthology.orchestra.run.smithi204.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-04-26T19:56:19.960 INFO:teuthology.orchestra.run.smithi204.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-26T19:56:19.993 INFO:teuthology.orchestra.run.smithi204.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-26T19:56:20.027 INFO:teuthology.orchestra.run.smithi204.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 951 kB/s | 64 kB 00:00 2024-04-26T19:56:20.052 INFO:teuthology.orchestra.run.smithi094.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 117 kB/s | 16 kB 00:00 2024-04-26T19:56:20.069 INFO:teuthology.orchestra.run.smithi204.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 595 kB/s | 45 kB 00:00 2024-04-26T19:56:20.527 INFO:teuthology.orchestra.run.smithi094.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 576 kB/s | 273 kB 00:00 2024-04-26T19:56:20.602 INFO:teuthology.orchestra.run.smithi204.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-04-26T19:56:20.930 INFO:teuthology.orchestra.run.smithi204.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-26T19:56:21.011 INFO:teuthology.orchestra.run.smithi204.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 3.7 MB/s | 309 kB 00:00 2024-04-26T19:56:21.061 INFO:teuthology.orchestra.run.smithi204.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-04-26T19:56:21.245 INFO:teuthology.orchestra.run.smithi204.stdout:(106/152): libarrow-9.0.0-11.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-04-26T19:56:21.295 INFO:teuthology.orchestra.run.smithi204.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-04-26T19:56:21.346 INFO:teuthology.orchestra.run.smithi204.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-04-26T19:56:21.404 INFO:teuthology.orchestra.run.smithi204.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-26T19:56:21.463 INFO:teuthology.orchestra.run.smithi204.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-26T19:56:21.530 INFO:teuthology.orchestra.run.smithi204.stdout:(111/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-04-26T19:56:21.536 INFO:teuthology.orchestra.run.smithi094.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 757 kB/s | 762 kB 00:01 2024-04-26T19:56:21.562 INFO:teuthology.orchestra.run.smithi103.stdout:(151/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 835 kB/s | 3.1 MB 00:03 2024-04-26T19:56:21.589 INFO:teuthology.orchestra.run.smithi204.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-04-26T19:56:21.639 INFO:teuthology.orchestra.run.smithi204.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 870 kB/s | 43 kB 00:00 2024-04-26T19:56:21.645 INFO:teuthology.orchestra.run.smithi094.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 170 kB/s | 18 kB 00:00 2024-04-26T19:56:21.681 INFO:teuthology.orchestra.run.smithi204.stdout:(114/152): protobuf-compiler-3.14.0-13.el9.x86_ 536 kB/s | 863 kB 00:01 2024-04-26T19:56:21.706 INFO:teuthology.orchestra.run.smithi204.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-04-26T19:56:21.765 INFO:teuthology.orchestra.run.smithi204.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-26T19:56:21.787 INFO:teuthology.orchestra.run.smithi094.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 982 kB/s | 7.4 MB 00:07 2024-04-26T19:56:21.790 INFO:teuthology.orchestra.run.smithi204.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-04-26T19:56:21.823 INFO:teuthology.orchestra.run.smithi204.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-04-26T19:56:21.874 INFO:teuthology.orchestra.run.smithi204.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 706 kB/s | 35 kB 00:00 2024-04-26T19:56:21.896 INFO:teuthology.orchestra.run.smithi094.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 322 kB/s | 80 kB 00:00 2024-04-26T19:56:21.991 INFO:teuthology.orchestra.run.smithi204.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.5 MB/s | 303 kB 00:00 2024-04-26T19:56:21.996 INFO:teuthology.orchestra.run.smithi094.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-04-26T19:56:22.016 INFO:teuthology.orchestra.run.smithi204.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.4 MB/s | 206 kB 00:00 2024-04-26T19:56:22.030 INFO:teuthology.orchestra.run.smithi094.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-04-26T19:56:22.063 INFO:teuthology.orchestra.run.smithi094.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.8 MB/s | 253 kB 00:00 2024-04-26T19:56:22.075 INFO:teuthology.orchestra.run.smithi204.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-26T19:56:22.114 INFO:teuthology.orchestra.run.smithi094.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-26T19:56:22.125 INFO:teuthology.orchestra.run.smithi204.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-04-26T19:56:22.181 INFO:teuthology.orchestra.run.smithi094.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 8.2 MB/s | 1.2 MB 00:00 2024-04-26T19:56:22.184 INFO:teuthology.orchestra.run.smithi204.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-26T19:56:22.206 INFO:teuthology.orchestra.run.smithi094.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-04-26T19:56:22.209 INFO:teuthology.orchestra.run.smithi204.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-04-26T19:56:22.231 INFO:teuthology.orchestra.run.smithi094.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.5 MB/s | 125 kB 00:00 2024-04-26T19:56:22.234 INFO:teuthology.orchestra.run.smithi204.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-04-26T19:56:22.259 INFO:teuthology.orchestra.run.smithi204.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-26T19:56:22.265 INFO:teuthology.orchestra.run.smithi094.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-26T19:56:22.301 INFO:teuthology.orchestra.run.smithi204.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 294 kB/s | 20 kB 00:00 2024-04-26T19:56:22.344 INFO:teuthology.orchestra.run.smithi204.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-04-26T19:56:22.361 INFO:teuthology.orchestra.run.smithi204.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 792 kB/s | 46 kB 00:00 2024-04-26T19:56:22.402 INFO:teuthology.orchestra.run.smithi204.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-26T19:56:22.412 INFO:teuthology.orchestra.run.smithi103.stdout:(152/152): librados2-19.0.0-3306.gbc612ffb.el9. 734 kB/s | 3.4 MB 00:04 2024-04-26T19:56:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-26T19:56:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:Total 4.5 MB/s | 222 MB 00:49 2024-04-26T19:56:22.428 INFO:teuthology.orchestra.run.smithi204.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 863 kB/s | 58 kB 00:00 2024-04-26T19:56:22.461 INFO:teuthology.orchestra.run.smithi204.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-04-26T19:56:22.478 INFO:teuthology.orchestra.run.smithi204.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-04-26T19:56:22.520 INFO:teuthology.orchestra.run.smithi204.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-26T19:56:22.545 INFO:teuthology.orchestra.run.smithi204.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 423 kB/s | 28 kB 00:00 2024-04-26T19:56:22.555 INFO:teuthology.orchestra.run.smithi094.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-04-26T19:56:22.571 INFO:teuthology.orchestra.run.smithi204.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-04-26T19:56:22.605 INFO:teuthology.orchestra.run.smithi204.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-26T19:56:22.616 INFO:teuthology.orchestra.run.smithi094.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-04-26T19:56:22.630 INFO:teuthology.orchestra.run.smithi204.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-26T19:56:22.649 INFO:teuthology.orchestra.run.smithi094.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-04-26T19:56:22.655 INFO:teuthology.orchestra.run.smithi204.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 711 kB/s | 36 kB 00:00 2024-04-26T19:56:22.666 INFO:teuthology.orchestra.run.smithi094.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 434 kB/s | 21 kB 00:00 2024-04-26T19:56:22.680 INFO:teuthology.orchestra.run.smithi204.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-26T19:56:22.714 INFO:teuthology.orchestra.run.smithi204.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-04-26T19:56:22.717 INFO:teuthology.orchestra.run.smithi094.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-04-26T19:56:22.740 INFO:teuthology.orchestra.run.smithi204.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-26T19:56:22.742 INFO:teuthology.orchestra.run.smithi094.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 437 kB/s | 33 kB 00:00 2024-04-26T19:56:22.773 INFO:teuthology.orchestra.run.smithi204.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-04-26T19:56:22.798 INFO:teuthology.orchestra.run.smithi204.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 342 kB/s | 20 kB 00:00 2024-04-26T19:56:22.801 INFO:teuthology.orchestra.run.smithi094.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 315 kB/s | 18 kB 00:00 2024-04-26T19:56:22.832 INFO:teuthology.orchestra.run.smithi204.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-26T19:56:22.868 INFO:teuthology.orchestra.run.smithi094.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 169 kB/s | 11 kB 00:00 2024-04-26T19:56:22.882 INFO:teuthology.orchestra.run.smithi204.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-04-26T19:56:22.960 INFO:teuthology.orchestra.run.smithi094.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-26T19:56:22.993 INFO:teuthology.orchestra.run.smithi094.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-26T19:56:23.019 INFO:teuthology.orchestra.run.smithi094.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 785 kB/s | 45 kB 00:00 2024-04-26T19:56:23.078 INFO:teuthology.orchestra.run.smithi094.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-04-26T19:56:23.128 INFO:teuthology.orchestra.run.smithi094.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 582 kB 00:00 2024-04-26T19:56:23.153 INFO:teuthology.orchestra.run.smithi094.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 342 kB/s | 25 kB 00:00 2024-04-26T19:56:23.204 INFO:teuthology.orchestra.run.smithi094.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 636 kB/s | 47 kB 00:00 2024-04-26T19:56:23.246 INFO:teuthology.orchestra.run.smithi094.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-04-26T19:56:23.279 INFO:teuthology.orchestra.run.smithi094.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-04-26T19:56:23.305 INFO:teuthology.orchestra.run.smithi094.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 610 kB/s | 35 kB 00:00 2024-04-26T19:56:23.480 INFO:teuthology.orchestra.run.smithi094.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-04-26T19:56:23.523 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-26T19:56:23.605 INFO:teuthology.orchestra.run.smithi094.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-26T19:56:23.669 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-26T19:56:23.669 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-26T19:56:23.681 INFO:teuthology.orchestra.run.smithi094.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 8.0 MB/s | 1.6 MB 00:00 2024-04-26T19:56:23.731 INFO:teuthology.orchestra.run.smithi094.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 7.2 MB/s | 911 kB 00:00 2024-04-26T19:56:23.790 INFO:teuthology.orchestra.run.smithi094.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-26T19:56:23.890 INFO:teuthology.orchestra.run.smithi094.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-26T19:56:23.949 INFO:teuthology.orchestra.run.smithi094.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T19:56:24.049 INFO:teuthology.orchestra.run.smithi094.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 18 MB/s | 5.8 MB 00:00 2024-04-26T19:56:24.116 INFO:teuthology.orchestra.run.smithi094.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-26T19:56:24.183 INFO:teuthology.orchestra.run.smithi094.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.7 MB/s | 248 kB 00:00 2024-04-26T19:56:24.292 INFO:teuthology.orchestra.run.smithi094.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-26T19:56:24.317 INFO:teuthology.orchestra.run.smithi094.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 193 kB/s | 26 kB 00:00 2024-04-26T19:56:24.409 INFO:teuthology.orchestra.run.smithi094.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 165 kB/s | 19 kB 00:00 2024-04-26T19:56:24.443 INFO:teuthology.orchestra.run.smithi094.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-04-26T19:56:24.468 INFO:teuthology.orchestra.run.smithi094.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 602 kB/s | 35 kB 00:00 2024-04-26T19:56:24.535 INFO:teuthology.orchestra.run.smithi094.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-26T19:56:24.602 INFO:teuthology.orchestra.run.smithi094.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-04-26T19:56:24.736 INFO:teuthology.orchestra.run.smithi094.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-04-26T19:56:24.761 INFO:teuthology.orchestra.run.smithi094.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 487 kB/s | 77 kB 00:00 2024-04-26T19:56:24.795 INFO:teuthology.orchestra.run.smithi094.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 732 kB/s | 42 kB 00:00 2024-04-26T19:56:24.829 INFO:teuthology.orchestra.run.smithi094.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 269 kB 00:00 2024-04-26T19:56:24.854 INFO:teuthology.orchestra.run.smithi094.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-26T19:56:24.938 INFO:teuthology.orchestra.run.smithi094.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 612 kB/s | 51 kB 00:00 2024-04-26T19:56:24.980 INFO:teuthology.orchestra.run.smithi094.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.8 MB/s | 279 kB 00:00 2024-04-26T19:56:25.013 INFO:teuthology.orchestra.run.smithi094.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 717 kB/s | 54 kB 00:00 2024-04-26T19:56:25.072 INFO:teuthology.orchestra.run.smithi094.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 722 kB/s | 42 kB 00:00 2024-04-26T19:56:25.131 INFO:teuthology.orchestra.run.smithi094.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 767 kB/s | 44 kB 00:00 2024-04-26T19:56:25.198 INFO:teuthology.orchestra.run.smithi094.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.3 MB/s | 221 kB 00:00 2024-04-26T19:56:25.282 INFO:teuthology.orchestra.run.smithi094.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 434 kB/s | 36 kB 00:00 2024-04-26T19:56:25.374 INFO:teuthology.orchestra.run.smithi094.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 522 kB/s | 47 kB 00:00 2024-04-26T19:56:25.438 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-26T19:56:25.438 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-26T19:56:25.440 INFO:teuthology.orchestra.run.smithi094.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-26T19:56:25.508 INFO:teuthology.orchestra.run.smithi094.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 873 kB/s | 58 kB 00:00 2024-04-26T19:56:25.524 INFO:teuthology.orchestra.run.smithi204.stdout:(148/152): librados2-19.0.0-3306.gbc612ffb.el9. 1.3 MB/s | 3.4 MB 00:02 2024-04-26T19:56:25.574 INFO:teuthology.orchestra.run.smithi204.stdout:(149/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 1.2 MB/s | 3.1 MB 00:02 2024-04-26T19:56:25.750 INFO:teuthology.orchestra.run.smithi094.stdout:(98/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-04-26T19:56:25.766 INFO:teuthology.orchestra.run.smithi204.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 5.6 MB/s | 1.3 MB 00:00 2024-04-26T19:56:25.775 INFO:teuthology.orchestra.run.smithi094.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 352 kB/s | 94 kB 00:00 2024-04-26T19:56:25.808 INFO:teuthology.orchestra.run.smithi094.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-26T19:56:25.834 INFO:teuthology.orchestra.run.smithi094.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-04-26T19:56:25.867 INFO:teuthology.orchestra.run.smithi094.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 383 kB/s | 22 kB 00:00 2024-04-26T19:56:25.919 INFO:teuthology.orchestra.run.smithi094.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 10 MB/s | 863 kB 00:00 2024-04-26T19:56:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.2 MB/s | 2.4 MB 00:00 2024-04-26T19:56:26.202 INFO:teuthology.orchestra.run.smithi094.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-26T19:56:26.236 INFO:teuthology.orchestra.run.smithi094.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-04-26T19:56:26.286 INFO:teuthology.orchestra.run.smithi094.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-04-26T19:56:26.345 INFO:teuthology.orchestra.run.smithi094.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 452 kB/s | 26 kB 00:00 2024-04-26T19:56:26.412 INFO:teuthology.orchestra.run.smithi094.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 737 kB/s | 49 kB 00:00 2024-04-26T19:56:26.521 INFO:teuthology.orchestra.run.smithi094.stdout:(109/152): libarrow-9.0.0-11.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-26T19:56:26.546 INFO:teuthology.orchestra.run.smithi094.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 506 kB/s | 67 kB 00:00 2024-04-26T19:56:26.588 INFO:teuthology.orchestra.run.smithi094.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-26T19:56:26.680 INFO:teuthology.orchestra.run.smithi094.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.2 MB/s | 839 kB 00:00 2024-04-26T19:56:26.730 INFO:teuthology.orchestra.run.smithi094.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-04-26T19:56:26.764 INFO:teuthology.orchestra.run.smithi094.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 524 kB/s | 43 kB 00:00 2024-04-26T19:56:26.798 INFO:teuthology.orchestra.run.smithi094.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 485 kB/s | 32 kB 00:00 2024-04-26T19:56:26.840 INFO:teuthology.orchestra.run.smithi094.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-04-26T19:56:26.882 INFO:teuthology.orchestra.run.smithi094.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-26T19:56:26.916 INFO:teuthology.orchestra.run.smithi094.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-26T19:56:26.974 INFO:teuthology.orchestra.run.smithi094.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.3 MB/s | 303 kB 00:00 2024-04-26T19:56:27.000 INFO:teuthology.orchestra.run.smithi094.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 425 kB/s | 35 kB 00:00 2024-04-26T19:56:27.050 INFO:teuthology.orchestra.run.smithi094.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-26T19:56:27.125 INFO:teuthology.orchestra.run.smithi094.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-26T19:56:27.192 INFO:teuthology.orchestra.run.smithi094.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-26T19:56:27.217 INFO:teuthology.orchestra.run.smithi094.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 117 kB/s | 11 kB 00:00 2024-04-26T19:56:27.243 INFO:teuthology.orchestra.run.smithi094.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 357 kB/s | 18 kB 00:00 2024-04-26T19:56:27.302 INFO:teuthology.orchestra.run.smithi094.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 279 kB/s | 23 kB 00:00 2024-04-26T19:56:27.318 INFO:teuthology.orchestra.run.smithi094.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 260 kB/s | 19 kB 00:00 2024-04-26T19:56:27.352 INFO:teuthology.orchestra.run.smithi094.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 397 kB/s | 20 kB 00:00 2024-04-26T19:56:27.411 INFO:teuthology.orchestra.run.smithi094.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-26T19:56:27.436 INFO:teuthology.orchestra.run.smithi094.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-04-26T19:56:27.470 INFO:teuthology.orchestra.run.smithi094.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-26T19:56:27.496 INFO:teuthology.orchestra.run.smithi094.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 994 kB/s | 58 kB 00:00 2024-04-26T19:56:27.538 INFO:teuthology.orchestra.run.smithi094.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-26T19:56:27.563 INFO:teuthology.orchestra.run.smithi094.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 247 kB/s | 16 kB 00:00 2024-04-26T19:56:27.597 INFO:teuthology.orchestra.run.smithi094.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-26T19:56:27.630 INFO:teuthology.orchestra.run.smithi094.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 430 kB/s | 28 kB 00:00 2024-04-26T19:56:27.664 INFO:teuthology.orchestra.run.smithi094.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 462 kB/s | 31 kB 00:00 2024-04-26T19:56:27.714 INFO:teuthology.orchestra.run.smithi094.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-26T19:56:27.740 INFO:teuthology.orchestra.run.smithi094.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 792 kB/s | 59 kB 00:00 2024-04-26T19:56:27.773 INFO:teuthology.orchestra.run.smithi094.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 618 kB/s | 36 kB 00:00 2024-04-26T19:56:27.807 INFO:teuthology.orchestra.run.smithi094.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 774 kB/s | 51 kB 00:00 2024-04-26T19:56:27.859 INFO:teuthology.orchestra.run.smithi094.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-26T19:56:27.891 INFO:teuthology.orchestra.run.smithi094.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-04-26T19:56:27.933 INFO:teuthology.orchestra.run.smithi094.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.7 MB/s | 427 kB 00:00 2024-04-26T19:56:27.958 INFO:teuthology.orchestra.run.smithi094.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 303 kB/s | 20 kB 00:00 2024-04-26T19:56:27.992 INFO:teuthology.orchestra.run.smithi094.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-26T19:56:28.067 INFO:teuthology.orchestra.run.smithi094.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-26T19:56:30.608 INFO:teuthology.orchestra.run.smithi204.stdout:(152/152): ceph-test-19.0.0-3306.gbc612ffb.el9. 1.1 MB/s | 49 MB 00:45 2024-04-26T19:56:30.611 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-04-26T19:56:30.611 INFO:teuthology.orchestra.run.smithi204.stdout:Total 3.9 MB/s | 222 MB 00:57 2024-04-26T19:56:30.734 INFO:teuthology.orchestra.run.smithi094.stdout:(148/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 1.2 MB/s | 3.1 MB 00:02 2024-04-26T19:56:30.926 INFO:teuthology.orchestra.run.smithi094.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 7.1 MB/s | 1.3 MB 00:00 2024-04-26T19:56:31.060 INFO:teuthology.orchestra.run.smithi094.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 18 MB/s | 2.4 MB 00:00 2024-04-26T19:56:31.464 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-26T19:56:31.558 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T19:56:31.596 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T19:56:31.635 INFO:teuthology.orchestra.run.smithi094.stdout:(151/152): librados2-19.0.0-3306.gbc612ffb.el9. 955 kB/s | 3.4 MB 00:03 2024-04-26T19:56:31.658 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T19:56:31.692 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T19:56:31.722 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-04-26T19:56:31.726 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T19:56:31.768 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T19:56:31.868 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-04-26T19:56:31.869 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-04-26T19:56:32.090 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T19:56:32.202 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T19:56:32.307 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T19:56:32.345 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T19:56:32.436 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T19:56:32.474 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T19:56:32.624 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T19:56:32.671 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T19:56:32.699 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T19:56:32.802 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T19:56:32.825 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T19:56:32.871 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T19:56:32.893 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T19:56:32.917 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T19:56:33.110 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T19:56:33.143 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T19:56:33.250 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T19:56:33.346 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T19:56:33.383 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T19:56:33.418 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T19:56:33.483 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T19:56:33.517 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T19:56:33.561 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T19:56:33.593 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T19:56:33.621 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T19:56:33.636 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-04-26T19:56:33.640 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-04-26T19:56:33.715 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T19:56:33.769 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T19:56:33.808 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T19:56:33.843 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T19:56:33.919 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T19:56:33.954 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T19:56:33.997 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T19:56:34.026 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T19:56:34.095 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T19:56:34.141 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T19:56:34.215 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T19:56:34.358 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T19:56:34.410 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T19:56:34.936 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T19:56:35.022 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T19:56:35.051 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T19:56:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T19:56:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T19:56:35.166 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T19:56:35.227 INFO:teuthology.orchestra.run.smithi094.stdout:(152/152): ceph-test-19.0.0-3306.gbc612ffb.el9. 1.4 MB/s | 49 MB 00:36 2024-04-26T19:56:35.230 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-26T19:56:35.231 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.9 MB/s | 222 MB 00:56 2024-04-26T19:56:35.318 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T19:56:35.369 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T19:56:35.424 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T19:56:35.496 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T19:56:35.536 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T19:56:35.573 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T19:56:35.619 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T19:56:35.656 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T19:56:35.698 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T19:56:35.740 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T19:56:35.876 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T19:56:35.913 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T19:56:35.943 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T19:56:35.993 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T19:56:36.087 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T19:56:36.135 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T19:56:36.175 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T19:56:36.201 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T19:56:36.225 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T19:56:36.300 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T19:56:36.427 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T19:56:36.458 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T19:56:36.486 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T19:56:36.520 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T19:56:36.559 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T19:56:36.591 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T19:56:36.621 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T19:56:36.656 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T19:56:36.689 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T19:56:36.761 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T19:56:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T19:56:37.110 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T19:56:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T19:56:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T19:56:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T19:56:37.338 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T19:56:37.387 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T19:56:38.017 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-04-26T19:56:38.104 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T19:56:38.159 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T19:56:38.193 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T19:56:38.258 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-04-26T19:56:38.258 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-04-26T19:56:38.295 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T19:56:38.351 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T19:56:38.438 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T19:56:38.470 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T19:56:38.648 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T19:56:38.676 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T19:56:38.754 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T19:56:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T19:56:39.516 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T19:56:39.540 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-04-26T19:56:39.628 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T19:56:39.675 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T19:56:39.728 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T19:56:39.762 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T19:56:39.788 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T19:56:39.830 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T19:56:40.167 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T19:56:40.280 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T19:56:40.384 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T19:56:40.415 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T19:56:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T19:56:40.536 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T19:56:40.689 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T19:56:40.732 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T19:56:40.737 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T19:56:40.769 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T19:56:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T19:56:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T19:56:40.864 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T19:56:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T19:56:40.887 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T19:56:40.896 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-04-26T19:56:40.896 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-04-26T19:56:40.933 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T19:56:40.955 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T19:56:40.978 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T19:56:41.178 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T19:56:41.179 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T19:56:41.213 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T19:56:41.222 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T19:56:41.314 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T19:56:41.319 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T19:56:41.343 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T19:56:41.379 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T19:56:41.417 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T19:56:41.453 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T19:56:41.489 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T19:56:41.562 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T19:56:41.613 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T19:56:41.657 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T19:56:41.689 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T19:56:41.717 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T19:56:41.788 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T19:56:41.856 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T19:56:41.866 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T19:56:41.892 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T19:56:41.904 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T19:56:41.947 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T19:56:41.963 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T19:56:41.993 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T19:56:42.025 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T19:56:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T19:56:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T19:56:42.130 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T19:56:42.201 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T19:56:42.254 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T19:56:42.329 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T19:56:42.474 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T19:56:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T19:56:43.058 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T19:56:43.144 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T19:56:43.172 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T19:56:43.215 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T19:56:43.253 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T19:56:43.298 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T19:56:43.441 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T19:56:43.506 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T19:56:43.561 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T19:56:43.635 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T19:56:43.665 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T19:56:43.702 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T19:56:43.749 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T19:56:43.785 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T19:56:43.828 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T19:56:43.870 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T19:56:44.007 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T19:56:44.059 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T19:56:44.089 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T19:56:44.131 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T19:56:44.226 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T19:56:44.264 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T19:56:44.304 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T19:56:44.330 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T19:56:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T19:56:44.354 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T19:56:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T19:56:44.422 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T19:56:44.566 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T19:56:44.604 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T19:56:44.632 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T19:56:44.667 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T19:56:44.705 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T19:56:44.737 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T19:56:44.767 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T19:56:44.802 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T19:56:44.835 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T19:56:44.877 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T19:56:44.903 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T19:56:44.922 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T19:56:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T19:56:44.990 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T19:56:45.041 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T19:56:45.057 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T19:56:45.057 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T19:56:45.057 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T19:56:45.057 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:56:45.073 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T19:56:45.088 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T19:56:45.120 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T19:56:45.219 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T19:56:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T19:56:46.007 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T19:56:46.055 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T19:56:46.089 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T19:56:46.191 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T19:56:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T19:56:46.327 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T19:56:46.425 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T19:56:46.630 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T19:56:46.655 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T19:56:46.734 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T19:56:47.257 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-04-26T19:56:47.384 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T19:56:47.403 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T19:56:47.502 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T19:56:47.507 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T19:56:47.624 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T19:56:47.689 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T19:56:47.728 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T19:56:47.877 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T19:56:48.323 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T19:56:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T19:56:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T19:56:48.644 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T19:56:48.742 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T19:56:48.753 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T19:56:48.793 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T19:56:48.828 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T19:56:48.855 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T19:56:48.889 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T19:56:49.123 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T19:56:49.183 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T19:56:49.201 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T19:56:49.206 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T19:56:49.243 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T19:56:49.332 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T19:56:49.364 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T19:56:49.365 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T19:56:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T19:56:49.401 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T19:56:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T19:56:49.560 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T19:56:49.631 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T19:56:49.879 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T19:56:49.913 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T19:56:49.918 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T19:56:49.967 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T19:56:49.986 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T19:56:49.989 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T19:56:50.187 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T19:56:50.311 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T19:56:50.362 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T19:56:50.409 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T19:56:50.511 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T19:56:50.556 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T19:56:50.614 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T19:56:50.660 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T19:56:50.691 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T19:56:50.800 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T19:56:50.870 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T19:56:50.904 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T19:56:50.958 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T19:56:51.064 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T19:56:51.107 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T19:56:51.161 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T19:56:51.198 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T19:56:51.311 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T19:56:51.393 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T19:56:51.492 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T19:56:51.675 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T19:56:51.737 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T19:56:52.279 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T19:56:52.320 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T19:56:52.379 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T19:56:52.485 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T19:56:52.522 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T19:56:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T19:56:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T19:56:52.716 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T19:56:52.861 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T19:56:52.898 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T19:56:52.977 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T19:56:52.977 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T19:56:52.977 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T19:56:52.977 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:56:52.994 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T19:56:53.008 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T19:56:53.070 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T19:56:53.154 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T19:56:53.283 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T19:56:53.327 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T19:56:53.366 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T19:56:53.435 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T19:56:53.482 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T19:56:53.543 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T19:56:53.601 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T19:56:53.797 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T19:56:53.877 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T19:56:53.920 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T19:56:53.978 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T19:56:54.126 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T19:56:54.168 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T19:56:54.219 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T19:56:54.257 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T19:56:54.292 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T19:56:54.497 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T19:56:54.680 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T19:56:54.742 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T19:56:54.782 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T19:56:54.828 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T19:56:54.895 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T19:56:54.955 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T19:56:55.005 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T19:56:55.109 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T19:56:55.161 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T19:56:55.246 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T19:56:55.317 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T19:56:55.462 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T19:56:55.518 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T19:56:55.564 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T19:56:55.633 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T19:56:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T19:56:55.843 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T19:56:56.757 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T19:56:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-04-26T19:56:56.757 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-04-26T19:56:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-04-26T19:56:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-04-26T19:56:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-04-26T19:56:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-04-26T19:56:56.758 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-04-26T19:56:56.758 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:56:56.784 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T19:56:56.919 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T19:56:56.926 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T19:56:56.942 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T19:56:56.942 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:56:56.942 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T19:56:56.942 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:56:56.963 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T19:56:57.096 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T19:56:57.156 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T19:56:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T19:56:57.305 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T19:56:57.537 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T19:56:57.568 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T19:56:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T19:56:58.438 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T19:56:58.464 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T19:56:58.465 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:56:58.465 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T19:56:58.465 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T19:56:58.465 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T19:56:58.465 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:56:58.480 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T19:56:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T19:56:58.689 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T19:56:58.711 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T19:56:58.712 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:56:58.712 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T19:56:58.712 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T19:56:58.712 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T19:56:58.712 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:56:59.148 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T19:56:59.171 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T19:56:59.171 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:56:59.171 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T19:56:59.171 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T19:56:59.171 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T19:56:59.171 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:56:59.279 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T19:56:59.337 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T19:56:59.416 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T19:56:59.609 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T19:56:59.657 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T19:56:59.735 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T19:56:59.824 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T19:56:59.963 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T19:57:00.008 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T19:57:00.040 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T19:57:00.067 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T19:57:00.192 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T19:57:00.287 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T19:57:00.336 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T19:57:00.407 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T19:57:00.478 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T19:57:00.508 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T19:57:00.540 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T19:57:00.609 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T19:57:00.639 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T19:57:00.802 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T19:57:00.842 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T19:57:00.986 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T19:57:01.023 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T19:57:01.089 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T19:57:01.640 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T19:57:01.670 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T19:57:01.687 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T19:57:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T19:57:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T19:57:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T19:57:02.665 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T19:57:02.692 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T19:57:02.791 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T19:57:02.945 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T19:57:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T19:57:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T19:57:02.992 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:02.992 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T19:57:02.992 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T19:57:02.992 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T19:57:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:57:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T19:57:03.047 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T19:57:03.144 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T19:57:03.193 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T19:57:04.115 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T19:57:04.147 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T19:57:04.170 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T19:57:04.171 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:04.171 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T19:57:04.171 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T19:57:04.171 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T19:57:04.171 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:57:04.215 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T19:57:04.237 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T19:57:04.237 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:04.237 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T19:57:04.237 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:57:04.291 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T19:57:04.313 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T19:57:04.313 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:04.313 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T19:57:04.313 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T19:57:04.314 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T19:57:04.314 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:57:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T19:57:04.483 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-04-26T19:57:04.483 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-04-26T19:57:04.483 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-04-26T19:57:04.483 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-04-26T19:57:04.483 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-04-26T19:57:04.483 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-04-26T19:57:04.483 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-04-26T19:57:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T19:57:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T19:57:04.616 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T19:57:04.639 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T19:57:04.639 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:04.639 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T19:57:04.639 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T19:57:04.639 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T19:57:04.639 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:57:04.646 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T19:57:04.661 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T19:57:04.661 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:04.661 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T19:57:04.662 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T19:57:05.233 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T19:57:05.391 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T19:57:05.391 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T19:57:05.391 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T19:57:05.391 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:05.442 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T19:57:06.149 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T19:57:06.175 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T19:57:06.175 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:06.175 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T19:57:06.175 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T19:57:06.175 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T19:57:06.175 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:06.410 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T19:57:06.432 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T19:57:06.433 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:06.433 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T19:57:06.433 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T19:57:06.433 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T19:57:06.433 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:06.885 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T19:57:06.908 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T19:57:06.908 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:06.908 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T19:57:06.908 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T19:57:06.908 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T19:57:06.908 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:07.016 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T19:57:07.073 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T19:57:07.129 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T19:57:07.315 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T19:57:07.362 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T19:57:07.449 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T19:57:07.538 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T19:57:07.678 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T19:57:07.720 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T19:57:07.752 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T19:57:07.780 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T19:57:08.173 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T19:57:08.203 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T19:57:08.227 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T19:57:08.289 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T19:57:08.318 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T19:57:09.245 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T19:57:09.287 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T19:57:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T19:57:09.359 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T19:57:09.367 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T19:57:09.387 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T19:57:09.412 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T19:57:09.570 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T19:57:09.616 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T19:57:09.638 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T19:57:10.185 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T19:57:10.228 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T19:57:10.247 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T19:57:10.371 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T19:57:10.404 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T19:57:10.508 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T19:57:10.674 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T19:57:10.707 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T19:57:10.730 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T19:57:10.730 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:10.730 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T19:57:10.730 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T19:57:10.730 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T19:57:10.730 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:10.773 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T19:57:10.785 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T19:57:10.881 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T19:57:10.930 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T19:57:11.075 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T19:57:11.160 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T19:57:11.201 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T19:57:11.284 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T19:57:11.314 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T19:57:11.314 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T19:57:11.398 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T19:57:11.431 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T19:57:11.431 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T19:57:11.528 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T19:57:11.596 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T19:57:11.852 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T19:57:11.884 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T19:57:11.907 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T19:57:11.907 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:11.907 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T19:57:11.907 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T19:57:11.908 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T19:57:11.908 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:11.952 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T19:57:11.974 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T19:57:11.974 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:11.974 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T19:57:11.974 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:12.019 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T19:57:12.045 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T19:57:12.045 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:12.045 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T19:57:12.045 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T19:57:12.045 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T19:57:12.045 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:12.353 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T19:57:12.376 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T19:57:12.376 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:12.376 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T19:57:12.377 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T19:57:12.377 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T19:57:12.377 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:16.957 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T19:57:16.999 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T19:57:17.039 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T19:57:17.097 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T19:57:17.124 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T19:57:17.308 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T19:57:17.353 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T19:57:17.384 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T19:57:17.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T19:57:17.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T19:57:17.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T19:57:17.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T19:57:17.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T19:57:17.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T19:57:17.676 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T19:57:17.677 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T19:57:17.678 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T19:57:17.679 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T19:57:17.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T19:57:17.681 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T19:57:17.681 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T19:57:17.681 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T19:57:17.681 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T19:57:17.681 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T19:57:17.681 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T19:57:17.681 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T19:57:17.681 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T19:57:17.681 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T19:57:18.009 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T19:57:18.048 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T19:57:18.068 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T19:57:18.626 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T19:57:18.626 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-04-26T19:57:18.626 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-04-26T19:57:18.627 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-04-26T19:57:18.627 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-04-26T19:57:18.627 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-04-26T19:57:18.627 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-04-26T19:57:18.627 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-04-26T19:57:18.627 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:18.849 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T19:57:18.898 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T19:57:18.898 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:18.899 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T19:57:18.899 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:18.910 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T19:57:18.995 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T19:57:19.028 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T19:57:19.119 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T19:57:19.149 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T19:57:19.149 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T19:57:19.225 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T19:57:19.250 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T19:57:19.250 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T19:57:19.327 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T19:57:19.390 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T19:57:19.797 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T19:57:19.797 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.798 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.799 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T19:57:19.800 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T19:57:19.801 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T19:57:19.802 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T19:57:19.803 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T19:57:19.804 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T19:57:19.805 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T19:57:19.806 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T19:57:19.807 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T19:57:19.808 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T19:57:19.809 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T19:57:19.809 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T19:57:19.809 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-26T19:57:20.207 DEBUG:teuthology.parallel:result is None 2024-04-26T19:57:20.476 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T19:57:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T19:57:20.557 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:20.557 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T19:57:20.557 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T19:57:20.557 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T19:57:20.557 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:20.810 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T19:57:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T19:57:20.879 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:20.879 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T19:57:20.879 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T19:57:20.879 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T19:57:20.879 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:21.369 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T19:57:21.445 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T19:57:21.445 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:21.446 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T19:57:21.446 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T19:57:21.446 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T19:57:21.446 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T19:57:21.637 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T19:57:21.706 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T19:57:21.935 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T19:57:22.008 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T19:57:22.096 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T19:57:22.200 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T19:57:22.364 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T19:57:22.416 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T19:57:22.466 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T19:57:22.507 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T19:57:23.194 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T19:57:23.245 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T19:57:23.303 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T19:57:23.411 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T19:57:23.447 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T19:57:24.757 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T19:57:24.807 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T19:57:25.465 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T19:57:25.466 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T19:57:25.467 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T19:57:25.467 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T19:57:25.467 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T19:57:25.467 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T19:57:25.467 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T19:57:25.481 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T19:57:25.481 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T19:57:25.481 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T19:57:25.481 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T19:57:25.481 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T19:57:25.481 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T19:57:25.481 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T19:57:25.481 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T19:57:25.482 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T19:57:25.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T19:57:25.484 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T19:57:25.485 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T19:57:25.486 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T19:57:25.487 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T19:57:25.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T19:57:25.489 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T19:57:25.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T19:57:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T19:57:25.941 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T19:57:26.086 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T19:57:26.264 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T19:57:26.302 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T19:57:26.371 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T19:57:26.371 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:26.371 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T19:57:26.371 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T19:57:26.372 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T19:57:26.372 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T19:57:26.482 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T19:57:26.617 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T19:57:26.689 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T19:57:27.540 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T19:57:27.541 INFO:teuthology.orchestra.run.smithi204.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.542 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:27.543 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.544 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T19:57:27.545 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T19:57:27.546 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T19:57:27.547 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T19:57:27.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T19:57:27.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T19:57:27.550 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T19:57:27.551 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T19:57:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T19:57:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T19:57:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T19:57:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T19:57:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T19:57:27.552 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T19:57:27.552 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-04-26T19:57:27.697 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T19:57:27.739 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T19:57:27.807 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T19:57:27.808 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:27.808 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T19:57:27.808 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T19:57:27.808 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T19:57:27.808 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:27.857 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T19:57:27.924 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T19:57:27.924 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:27.924 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T19:57:27.925 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:27.957 DEBUG:teuthology.parallel:result is None 2024-04-26T19:57:28.010 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T19:57:28.082 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T19:57:28.083 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:28.083 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T19:57:28.083 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T19:57:28.083 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T19:57:28.083 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:28.442 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T19:57:28.511 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T19:57:28.511 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T19:57:28.512 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T19:57:28.512 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T19:57:28.512 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T19:57:28.512 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:33.530 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T19:57:33.586 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T19:57:33.679 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T19:57:33.722 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T19:57:33.769 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T19:57:34.103 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T19:57:34.156 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T19:57:34.183 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T19:57:35.088 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T19:57:35.158 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T19:57:35.213 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T19:57:36.516 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T19:57:37.036 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T19:57:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T19:57:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T19:57:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T19:57:37.356 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T19:57:37.516 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T19:57:37.563 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T19:57:37.564 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T19:57:37.712 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T19:57:37.845 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T19:57:38.119 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T19:57:38.120 INFO:teuthology.orchestra.run.smithi094.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T19:57:38.120 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:43.818 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T19:57:43.819 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T19:57:43.820 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T19:57:43.832 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T19:57:43.833 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T19:57:43.834 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T19:57:43.835 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T19:57:43.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T19:57:43.837 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T19:57:43.838 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T19:57:43.839 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T19:57:43.840 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T19:57:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T19:57:43.842 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T19:57:43.842 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T19:57:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.373 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T19:57:46.374 INFO:teuthology.orchestra.run.smithi094.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T19:57:46.375 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T19:57:46.376 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T19:57:46.377 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T19:57:46.378 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T19:57:46.379 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T19:57:46.380 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T19:57:46.381 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T19:57:46.382 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T19:57:46.383 INFO:teuthology.orchestra.run.smithi094.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T19:57:46.383 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:57:46.383 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-04-26T19:57:46.973 DEBUG:teuthology.parallel:result is None 2024-04-26T19:57:46.973 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:57:47.196 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T19:57:47.251 INFO:teuthology.orchestra.run.smithi094.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T19:57:47.251 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T19:57:47.251 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T19:57:47.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:57:47.474 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T19:57:47.500 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T19:57:47.500 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T19:57:47.500 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T19:57:47.502 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:57:47.714 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T19:57:47.739 INFO:teuthology.orchestra.run.smithi204.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T19:57:47.739 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T19:57:47.740 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T19:57:47.741 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T19:57:47.741 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T19:57:47.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T19:57:47.834 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T19:57:47.834 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T19:57:47.874 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T19:57:47.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T19:57:47.906 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T19:57:47.906 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T19:57:47.906 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T19:57:47.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T19:57:48.094 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T19:57:48.095 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T19:57:48.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T19:57:48.192 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T19:57:48.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T19:57:48.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T19:57:48.288 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T19:57:48.289 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T19:57:48.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T19:57:48.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T19:57:48.465 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T19:57:48.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T19:57:48.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T19:57:48.567 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T19:57:48.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T19:57:48.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T19:57:48.656 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T19:57:48.656 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T19:57:48.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T19:57:48.732 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T19:57:48.839 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T19:57:48.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T19:57:48.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T19:57:48.927 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T19:57:48.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T19:57:48.951 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T19:57:49.015 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T19:57:49.120 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2}} 2024-04-26T19:57:49.120 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:57:49.120 INFO:tasks.cephadm:Cluster fsid is 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:57:49.121 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T19:57:49.121 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.94', 'mon.b': '172.21.15.103', 'mon.c': '172.21.15.204'} 2024-04-26T19:57:49.121 INFO:tasks.cephadm:First mon is mon.a on smithi094 2024-04-26T19:57:49.121 INFO:tasks.cephadm:First mgr is x 2024-04-26T19:57:49.121 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T19:57:49.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo hostname $(hostname -s) 2024-04-26T19:57:49.185 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-04-26T19:57:49.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-04-26T19:57:49.231 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T19:57:49.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:57:49.617 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T19:57:49.750 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:57:49.751 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-26T19:57:49.752 DEBUG:teuthology.orchestra.run.smithi094:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T19:57:51.390 INFO:teuthology.orchestra.run.smithi094.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 19:57 /home/ubuntu/cephtest/cephadm 2024-04-26T19:57:51.390 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T19:57:52.122 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 19:57 /home/ubuntu/cephtest/cephadm 2024-04-26T19:57:52.123 DEBUG:teuthology.orchestra.run.smithi204:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T19:57:53.185 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 19:57 /home/ubuntu/cephtest/cephadm 2024-04-26T19:57:53.185 DEBUG:teuthology.orchestra.run.smithi094:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T19:57:53.229 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T19:57:53.248 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T19:57:53.283 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-26T19:57:53.283 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T19:57:53.288 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T19:57:53.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T19:57:54.898 INFO:teuthology.orchestra.run.smithi204.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T19:57:54.923 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T19:57:55.805 INFO:teuthology.orchestra.run.smithi094.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T19:58:24.350 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-04-26T19:58:24.350 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T19:58:24.350 INFO:teuthology.orchestra.run.smithi204.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T19:58:24.351 INFO:teuthology.orchestra.run.smithi204.stdout: "repo_digests": [ 2024-04-26T19:58:24.351 INFO:teuthology.orchestra.run.smithi204.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T19:58:24.351 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-04-26T19:58:24.351 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-04-26T19:58:42.605 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-26T19:58:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T19:58:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T19:58:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-04-26T19:58:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T19:58:42.606 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-26T19:58:42.606 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-26T19:58:45.680 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-26T19:58:45.681 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T19:58:45.681 INFO:teuthology.orchestra.run.smithi094.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T19:58:45.681 INFO:teuthology.orchestra.run.smithi094.stdout: "repo_digests": [ 2024-04-26T19:58:45.681 INFO:teuthology.orchestra.run.smithi094.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T19:58:45.681 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-04-26T19:58:45.681 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-26T19:58:45.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph 2024-04-26T19:58:45.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-04-26T19:58:45.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-04-26T19:58:45.874 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /etc/ceph 2024-04-26T19:58:45.962 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-04-26T19:58:45.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-04-26T19:58:46.020 INFO:tasks.cephadm:Writing seed config... 2024-04-26T19:58:46.021 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T19:58:46.021 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T19:58:46.021 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T19:58:46.021 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T19:58:46.021 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T19:58:46.021 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T19:58:46.021 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T19:58:46.021 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T19:58:46.021 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T19:58:46.022 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T19:58:46.023 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T19:58:46.024 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T19:58:46.025 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T19:58:46.025 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T19:58:46.080 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 420d0bd6-0407-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T19:58:46.081 DEBUG:teuthology.orchestra.run.smithi094:mon.a> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T19:58:46.128 DEBUG:teuthology.orchestra.run.smithi094:mgr.x> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T19:58:46.175 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T19:58:46.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.94 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T19:58:46.444 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-04-26T19:58:46.444 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', '420d0bd6-0407-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.94', '--skip-admin-label'] 2024-04-26T19:58:46.444 INFO:teuthology.orchestra.run.smithi094.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T19:58:46.445 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying podman|docker is present... 2024-04-26T19:58:46.498 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T19:58:46.498 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying lvm2 is present... 2024-04-26T19:58:46.499 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying time synchronization is in place... 2024-04-26T19:58:46.513 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T19:58:46.514 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T19:58:46.528 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T19:58:46.529 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-26T19:58:46.545 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout enabled 2024-04-26T19:58:46.563 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout active 2024-04-26T19:58:46.564 INFO:teuthology.orchestra.run.smithi094.stdout:Unit chronyd.service is enabled and running 2024-04-26T19:58:46.564 INFO:teuthology.orchestra.run.smithi094.stdout:Repeating the final host check... 2024-04-26T19:58:46.621 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T19:58:46.621 INFO:teuthology.orchestra.run.smithi094.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T19:58:46.621 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl is present 2024-04-26T19:58:46.621 INFO:teuthology.orchestra.run.smithi094.stdout:lvcreate is present 2024-04-26T19:58:46.637 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T19:58:46.637 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T19:58:46.655 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T19:58:46.656 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-26T19:58:46.678 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout enabled 2024-04-26T19:58:46.699 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout active 2024-04-26T19:58:46.699 INFO:teuthology.orchestra.run.smithi094.stdout:Unit chronyd.service is enabled and running 2024-04-26T19:58:46.700 INFO:teuthology.orchestra.run.smithi094.stdout:Host looks OK 2024-04-26T19:58:46.700 INFO:teuthology.orchestra.run.smithi094.stdout:Cluster fsid: 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:58:46.700 INFO:teuthology.orchestra.run.smithi094.stdout:Acquiring lock 140193859972256 on /run/cephadm/420d0bd6-0407-11ef-bc93-c7b262605968.lock 2024-04-26T19:58:46.701 INFO:teuthology.orchestra.run.smithi094.stdout:Lock 140193859972256 acquired on /run/cephadm/420d0bd6-0407-11ef-bc93-c7b262605968.lock 2024-04-26T19:58:46.702 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying IP 172.21.15.94 port 3300 ... 2024-04-26T19:58:46.703 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying IP 172.21.15.94 port 6789 ... 2024-04-26T19:58:46.703 INFO:teuthology.orchestra.run.smithi094.stdout:Base mon IP(s) is [172.21.15.94:3300, 172.21.15.94:6789], mon addrv is [v2:172.21.15.94:3300,v1:172.21.15.94:6789] 2024-04-26T19:58:46.717 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.94 metric 100 2024-04-26T19:58:46.717 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.94 metric 100 2024-04-26T19:58:46.730 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T19:58:46.730 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-26T19:58:46.730 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1773sec hoplimit 64 pref medium 2024-04-26T19:58:46.741 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T19:58:46.741 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T19:58:46.741 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T19:58:46.741 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-26T19:58:46.741 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d763/64 scope link 2024-04-26T19:58:46.741 INFO:teuthology.orchestra.run.smithi094.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T19:58:46.744 INFO:teuthology.orchestra.run.smithi094.stdout:Mon IP `172.21.15.94` is in CIDR network `172.21.0.0/20` 2024-04-26T19:58:46.744 INFO:teuthology.orchestra.run.smithi094.stdout:Mon IP `172.21.15.94` is in CIDR network `172.21.0.0/20` 2024-04-26T19:58:46.744 INFO:teuthology.orchestra.run.smithi094.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T19:58:46.746 INFO:teuthology.orchestra.run.smithi094.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T19:58:46.749 INFO:teuthology.orchestra.run.smithi094.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T19:58:47.459 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T19:58:47.459 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T19:58:47.459 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T19:58:47.459 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T19:58:47.460 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-26T19:58:47.460 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T19:58:47.460 INFO:teuthology.orchestra.run.smithi094.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T19:58:48.664 INFO:teuthology.orchestra.run.smithi094.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T19:58:48.665 INFO:teuthology.orchestra.run.smithi094.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T19:58:48.665 INFO:teuthology.orchestra.run.smithi094.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T19:58:49.745 INFO:teuthology.orchestra.run.smithi094.stdout:stat: stdout 167 167 2024-04-26T19:58:49.745 INFO:teuthology.orchestra.run.smithi094.stdout:Creating initial keys... 2024-04-26T19:58:50.917 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph-authtool: stdout AQB6ByxmDatbDxAAEipsUhnH75tjM38ufBOfeQ== 2024-04-26T19:58:52.021 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph-authtool: stdout AQB7ByxmHuMIFRAAAn72ZdapdFSCO8LeShIa/A== 2024-04-26T19:58:53.214 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph-authtool: stdout AQB8Byxm6j5pHRAAcPv/QpW3wiTQKne5Hr3olw== 2024-04-26T19:58:53.215 INFO:teuthology.orchestra.run.smithi094.stdout:Creating initial monmap... 2024-04-26T19:58:54.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T19:58:54.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T19:58:54.329 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:58:54.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T19:58:54.330 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool for a [v2:172.21.15.94:3300,v1:172.21.15.94:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T19:58:54.330 INFO:teuthology.orchestra.run.smithi094.stdout:setting min_mon_release = quincy 2024-04-26T19:58:54.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: set fsid to 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:58:54.330 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T19:58:54.330 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T19:58:54.330 INFO:teuthology.orchestra.run.smithi094.stdout:Creating mon... 2024-04-26T19:58:55.471 INFO:teuthology.orchestra.run.smithi094.stdout:create mon.a on 2024-04-26T19:58:55.871 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T19:58:56.115 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T19:58:56.422 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-420d0bd6-0407-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-420d0bd6-0407-11ef-bc93-c7b262605968.target. 2024-04-26T19:58:56.422 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-420d0bd6-0407-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-420d0bd6-0407-11ef-bc93-c7b262605968.target. 2024-04-26T19:58:56.805 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.a 2024-04-26T19:58:56.805 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to reset failed state of unit ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T19:58:57.094 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-420d0bd6-0407-11ef-bc93-c7b262605968.target.wants/ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-420d0bd6-0407-11ef-bc93-c7b262605968@.service. 2024-04-26T19:58:57.824 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:57 smithi094 podman[27664]: 2024-04-26 19:58:57.568726777 +0000 UTC m=+0.155166336 container create 313cdd3a840b870bf8c6714746e4082e77089eb69bb8953c8bd96ea38bee3a10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-26T19:58:57.824 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:57 smithi094 podman[27664]: 2024-04-26 19:58:57.822098042 +0000 UTC m=+0.408537637 container init 313cdd3a840b870bf8c6714746e4082e77089eb69bb8953c8bd96ea38bee3a10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T19:58:57.943 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T19:58:57.943 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-04-26T19:58:57.963 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T19:58:57.963 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-26T19:58:57.963 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-04-26T19:58:57.963 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T19:58:57.964 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mon to start... 2024-04-26T19:58:57.965 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mon... 2024-04-26T19:58:58.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:57 smithi094 podman[27664]: 2024-04-26 19:58:57.832256385 +0000 UTC m=+0.418695951 container start 313cdd3a840b870bf8c6714746e4082e77089eb69bb8953c8bd96ea38bee3a10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True) 2024-04-26T19:58:58.116 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:57 smithi094 ceph-mon[27682]: mkfs 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:58:58.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:57 smithi094 bash[27664]: 313cdd3a840b870bf8c6714746e4082e77089eb69bb8953c8bd96ea38bee3a10 2024-04-26T19:58:58.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:57 smithi094 ceph-mon[27682]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T19:58:58.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:57 smithi094 systemd[1]: Started Ceph mon.a for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: monmap epoch 1 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: last_changed 2024-04-26T19:58:53.653581+0000 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: created 2024-04-26T19:58:53.653581+0000 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: min_mon_release 19 (squid) 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: election_strategy: 1 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: fsmap 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: mgrmap e1: no daemons active 2024-04-26T19:58:58.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:58:58 smithi094 ceph-mon[27682]: from='client.? 172.21.15.94:0/1940100576' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T19:58:59.313 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T19:58:59.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout id: 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:58:59.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T19:58:59.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:58:59.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout services: 2024-04-26T19:58:59.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.768773s) 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout data: 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.581+0000 7f2d9f800700 1 Processor -- start 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.582+0000 7f2d9f800700 1 -- start start 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.582+0000 7f2d9f800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d98107a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.582+0000 7f2d9f800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9807cdb0 con 0x7f2d98107650 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.583+0000 7f2d9ce00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d98107a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:58:59.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.583+0000 7f2d9ce00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d98107a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54784/0 (socket says 172.21.15.94:54784) 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.583+0000 7f2d9ce00700 1 -- 172.21.15.94:0/4098589732 learned_addr learned my addr 172.21.15.94:0/4098589732 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.584+0000 7f2d9ce00700 1 -- 172.21.15.94:0/4098589732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d98107fb0 con 0x7f2d98107650 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.584+0000 7f2d9ce00700 1 --2- 172.21.15.94:0/4098589732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d98107a70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0098a0 tx=0x7f2d8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81948fe07acd3a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.584+0000 7f2d97400700 1 -- 172.21.15.94:0/4098589732 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c004160 con 0x7f2d98107650 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.584+0000 7f2d97400700 1 -- 172.21.15.94:0/4098589732 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d8c004300 con 0x7f2d98107650 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.584+0000 7f2d97400700 1 -- 172.21.15.94:0/4098589732 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c0036a0 con 0x7f2d98107650 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.585+0000 7f2d9f800700 1 -- 172.21.15.94:0/4098589732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 msgr2=0x7f2d98107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.585+0000 7f2d9f800700 1 --2- 172.21.15.94:0/4098589732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d98107a70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0098a0 tx=0x7f2d8c009bb0 comp rx=0 tx=0).stop 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.585+0000 7f2d9f800700 1 -- 172.21.15.94:0/4098589732 shutdown_connections 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.585+0000 7f2d9f800700 1 --2- 172.21.15.94:0/4098589732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d98107a70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.585+0000 7f2d9f800700 1 -- 172.21.15.94:0/4098589732 >> 172.21.15.94:0/4098589732 conn(0x7f2d98103230 msgr2=0x7f2d98105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.585+0000 7f2d9f800700 1 -- 172.21.15.94:0/4098589732 shutdown_connections 2024-04-26T19:58:59.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.585+0000 7f2d9f800700 1 -- 172.21.15.94:0/4098589732 wait complete. 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.586+0000 7f2d9f800700 1 Processor -- start 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.586+0000 7f2d9f800700 1 -- start start 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.587+0000 7f2d9f800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d9819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.587+0000 7f2d9f800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2d981084e0 con 0x7f2d98107650 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.587+0000 7f2d9ce00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d9819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.587+0000 7f2d9ce00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d9819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54794/0 (socket says 172.21.15.94:54794) 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.587+0000 7f2d9ce00700 1 -- 172.21.15.94:0/1940100576 learned_addr learned my addr 172.21.15.94:0/1940100576 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.587+0000 7f2d9ce00700 1 -- 172.21.15.94:0/1940100576 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9819eb90 con 0x7f2d98107650 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.588+0000 7f2d9ce00700 1 --2- 172.21.15.94:0/1940100576 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d9819e650 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c000c00 tx=0x7f2d8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.588+0000 7f2d95600700 1 -- 172.21.15.94:0/1940100576 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c003db0 con 0x7f2d98107650 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.588+0000 7f2d95600700 1 -- 172.21.15.94:0/1940100576 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d8c011630 con 0x7f2d98107650 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.588+0000 7f2d95600700 1 -- 172.21.15.94:0/1940100576 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c011950 con 0x7f2d98107650 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.588+0000 7f2d9f800700 1 -- 172.21.15.94:0/1940100576 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9819ee20 con 0x7f2d98107650 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.588+0000 7f2d9f800700 1 -- 172.21.15.94:0/1940100576 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d981a1ac0 con 0x7f2d98107650 2024-04-26T19:58:59.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.588+0000 7f2d95600700 1 -- 172.21.15.94:0/1940100576 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2d8c01a460 con 0x7f2d98107650 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.589+0000 7f2d95600700 1 -- 172.21.15.94:0/1940100576 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2d8c01b780 con 0x7f2d98107650 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.589+0000 7f2d9f800700 1 -- 172.21.15.94:0/1940100576 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d98064a80 con 0x7f2d98107650 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.593+0000 7f2d95600700 1 -- 172.21.15.94:0/1940100576 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f2d8c01b980 con 0x7f2d98107650 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.672+0000 7f2d9f800700 1 -- 172.21.15.94:0/1940100576 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f2d981a1c90 con 0x7f2d98107650 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.673+0000 7f2d95600700 1 -- 172.21.15.94:0/1940100576 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f2d8c01bb60 con 0x7f2d98107650 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.674+0000 7f2d9f800700 1 -- 172.21.15.94:0/1940100576 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 msgr2=0x7f2d9819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.674+0000 7f2d9f800700 1 --2- 172.21.15.94:0/1940100576 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d9819e650 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c000c00 tx=0x7f2d8c004750 comp rx=0 tx=0).stop 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.674+0000 7f2d9f800700 1 -- 172.21.15.94:0/1940100576 shutdown_connections 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.674+0000 7f2d9f800700 1 --2- 172.21.15.94:0/1940100576 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d98107650 0x7f2d9819e650 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.674+0000 7f2d9f800700 1 -- 172.21.15.94:0/1940100576 >> 172.21.15.94:0/1940100576 conn(0x7f2d98103230 msgr2=0x7f2d98194c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.674+0000 7f2d9f800700 1 -- 172.21.15.94:0/1940100576 shutdown_connections 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:58.675+0000 7f2d9f800700 1 -- 172.21.15.94:0/1940100576 wait complete. 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:mon is available 2024-04-26T19:58:59.324 INFO:teuthology.orchestra.run.smithi094.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T19:59:00.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:00 smithi094 ceph-mon[27682]: from='client.? 172.21.15.94:0/653619998' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T19:59:00.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:00 smithi094 ceph-mon[27682]: from='client.? 172.21.15.94:0/653619998' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T19:59:00.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:00.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T19:59:00.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout fsid = 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:59:00.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T19:59:00.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.94:3300,v1:172.21.15.94:6789] 2024-04-26T19:59:00.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T19:59:00.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T19:59:00.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T19:59:00.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T19:59:00.657 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.872+0000 7fcd54400700 1 Processor -- start 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.872+0000 7fcd54400700 1 -- start start 2024-04-26T19:59:00.658 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.873+0000 7fcd54400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c10b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:00.659 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.873+0000 7fcd54400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4c10b8c0 con 0x7fcd4c10af60 2024-04-26T19:59:00.659 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.873+0000 7fcd51a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c10b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:00.659 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.874+0000 7fcd51a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c10b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54804/0 (socket says 172.21.15.94:54804) 2024-04-26T19:59:00.659 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.874+0000 7fcd51a00700 1 -- 172.21.15.94:0/688614219 learned_addr learned my addr 172.21.15.94:0/688614219 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:00.659 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.874+0000 7fcd51a00700 1 -- 172.21.15.94:0/688614219 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4c10c070 con 0x7fcd4c10af60 2024-04-26T19:59:00.659 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.875+0000 7fcd51a00700 1 --2- 172.21.15.94:0/688614219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c10b380 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcd400098a0 tx=0x7fcd40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c54e31de0c3c873 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:00.659 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.875+0000 7fcd4be00700 1 -- 172.21.15.94:0/688614219 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd40004160 con 0x7fcd4c10af60 2024-04-26T19:59:00.662 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.875+0000 7fcd4be00700 1 -- 172.21.15.94:0/688614219 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd40004300 con 0x7fcd4c10af60 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.875+0000 7fcd4be00700 1 -- 172.21.15.94:0/688614219 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd400036a0 con 0x7fcd4c10af60 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.876+0000 7fcd54400700 1 -- 172.21.15.94:0/688614219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 msgr2=0x7fcd4c10b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.876+0000 7fcd54400700 1 --2- 172.21.15.94:0/688614219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c10b380 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcd400098a0 tx=0x7fcd40009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.876+0000 7fcd54400700 1 -- 172.21.15.94:0/688614219 shutdown_connections 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.876+0000 7fcd54400700 1 --2- 172.21.15.94:0/688614219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c10b380 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.876+0000 7fcd54400700 1 -- 172.21.15.94:0/688614219 >> 172.21.15.94:0/688614219 conn(0x7fcd4c07e300 msgr2=0x7fcd4c07e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.876+0000 7fcd54400700 1 -- 172.21.15.94:0/688614219 shutdown_connections 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.876+0000 7fcd54400700 1 -- 172.21.15.94:0/688614219 wait complete. 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.877+0000 7fcd54400700 1 Processor -- start 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.877+0000 7fcd54400700 1 -- start start 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.878+0000 7fcd54400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.878+0000 7fcd54400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4c10c540 con 0x7fcd4c10af60 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.878+0000 7fcd51a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:00.663 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.878+0000 7fcd51a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c19e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54806/0 (socket says 172.21.15.94:54806) 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.878+0000 7fcd51a00700 1 -- 172.21.15.94:0/653619998 learned_addr learned my addr 172.21.15.94:0/653619998 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.878+0000 7fcd51a00700 1 -- 172.21.15.94:0/653619998 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4c19eb40 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.879+0000 7fcd51a00700 1 --2- 172.21.15.94:0/653619998 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c19e600 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcd40000c00 tx=0x7fcd40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.879+0000 7fcd4a000700 1 -- 172.21.15.94:0/653619998 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd40003db0 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.879+0000 7fcd4a000700 1 -- 172.21.15.94:0/653619998 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd40011630 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.879+0000 7fcd54400700 1 -- 172.21.15.94:0/653619998 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd4c19edd0 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.879+0000 7fcd4a000700 1 -- 172.21.15.94:0/653619998 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd40011950 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.879+0000 7fcd54400700 1 -- 172.21.15.94:0/653619998 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd4c1a1ac0 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.879+0000 7fcd4a000700 1 -- 172.21.15.94:0/653619998 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcd4001a460 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.879+0000 7fcd4a000700 1 -- 172.21.15.94:0/653619998 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcd4001b780 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.880+0000 7fcd54400700 1 -- 172.21.15.94:0/653619998 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd4c064a80 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.883+0000 7fcd4a000700 1 -- 172.21.15.94:0/653619998 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fcd40004260 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.960+0000 7fcd54400700 1 -- 172.21.15.94:0/653619998 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fcd4c02e6e0 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.969+0000 7fcd4a000700 1 -- 172.21.15.94:0/653619998 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7fcd4001a750 con 0x7fcd4c10af60 2024-04-26T19:59:00.664 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.971+0000 7fcd4a000700 1 -- 172.21.15.94:0/653619998 <== mon.0 v2:172.21.15.94:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcd4001be10 con 0x7fcd4c10af60 2024-04-26T19:59:00.665 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.972+0000 7fcd54400700 1 -- 172.21.15.94:0/653619998 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 msgr2=0x7fcd4c19e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:00.665 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.972+0000 7fcd54400700 1 --2- 172.21.15.94:0/653619998 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c19e600 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcd40000c00 tx=0x7fcd40004750 comp rx=0 tx=0).stop 2024-04-26T19:59:00.665 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.972+0000 7fcd54400700 1 -- 172.21.15.94:0/653619998 shutdown_connections 2024-04-26T19:59:00.665 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.972+0000 7fcd54400700 1 --2- 172.21.15.94:0/653619998 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd4c10af60 0x7fcd4c19e600 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:00.665 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.972+0000 7fcd54400700 1 -- 172.21.15.94:0/653619998 >> 172.21.15.94:0/653619998 conn(0x7fcd4c07e300 msgr2=0x7fcd4c195170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:00.665 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.973+0000 7fcd54400700 1 -- 172.21.15.94:0/653619998 shutdown_connections 2024-04-26T19:59:00.665 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:58:59.973+0000 7fcd54400700 1 -- 172.21.15.94:0/653619998 wait complete. 2024-04-26T19:59:00.665 INFO:teuthology.orchestra.run.smithi094.stdout:Generating new minimal ceph.conf... 2024-04-26T19:59:01.652 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:01 smithi094 ceph-mon[27682]: from='client.? 172.21.15.94:0/3867653115' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T19:59:01.969 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.269+0000 7f48e9000700 1 Processor -- start 2024-04-26T19:59:01.969 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.269+0000 7f48e9000700 1 -- start start 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.269+0000 7f48e9000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e410b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.270+0000 7f48e9000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f48e410b780 con 0x7f48e410ad90 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.270+0000 7f48e3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e410b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.270+0000 7f48e3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e410b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52364/0 (socket says 172.21.15.94:52364) 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.270+0000 7f48e3400700 1 -- 172.21.15.94:0/3074662569 learned_addr learned my addr 172.21.15.94:0/3074662569 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.270+0000 7f48e3400700 1 -- 172.21.15.94:0/3074662569 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e410bf90 con 0x7f48e410ad90 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.271+0000 7f48e3400700 1 --2- 172.21.15.94:0/3074662569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e410b1b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f48d40098a0 tx=0x7f48d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e65c7d5fee0fa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.271+0000 7f48e2000700 1 -- 172.21.15.94:0/3074662569 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48d4004160 con 0x7f48e410ad90 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.271+0000 7f48e2000700 1 -- 172.21.15.94:0/3074662569 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f48d4004300 con 0x7f48e410ad90 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.271+0000 7f48e2000700 1 -- 172.21.15.94:0/3074662569 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48d4011af0 con 0x7f48e410ad90 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.272+0000 7f48e9000700 1 -- 172.21.15.94:0/3074662569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 msgr2=0x7f48e410b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.272+0000 7f48e9000700 1 --2- 172.21.15.94:0/3074662569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e410b1b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f48d40098a0 tx=0x7f48d4009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.272+0000 7f48e9000700 1 -- 172.21.15.94:0/3074662569 shutdown_connections 2024-04-26T19:59:01.970 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.272+0000 7f48e9000700 1 --2- 172.21.15.94:0/3074662569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e410b1b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:01.971 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.272+0000 7f48e9000700 1 -- 172.21.15.94:0/3074662569 >> 172.21.15.94:0/3074662569 conn(0x7f48e407e170 msgr2=0x7f48e407e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:01.971 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.272+0000 7f48e9000700 1 -- 172.21.15.94:0/3074662569 shutdown_connections 2024-04-26T19:59:01.971 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.272+0000 7f48e9000700 1 -- 172.21.15.94:0/3074662569 wait complete. 2024-04-26T19:59:01.971 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.273+0000 7f48e9000700 1 Processor -- start 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.273+0000 7f48e9000700 1 -- start start 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.273+0000 7f48e9000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.273+0000 7f48e9000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f48e410f1c0 con 0x7f48e410ad90 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.274+0000 7f48e3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.274+0000 7f48e3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e419ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52368/0 (socket says 172.21.15.94:52368) 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.274+0000 7f48e3400700 1 -- 172.21.15.94:0/3867653115 learned_addr learned my addr 172.21.15.94:0/3867653115 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.274+0000 7f48e3400700 1 -- 172.21.15.94:0/3867653115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e419f200 con 0x7f48e410ad90 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.274+0000 7f48e3400700 1 --2- 172.21.15.94:0/3867653115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e419ecc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48d40117b0 tx=0x7f48d4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.275+0000 7f48dbe00700 1 -- 172.21.15.94:0/3867653115 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48d4003af0 con 0x7f48e410ad90 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.275+0000 7f48dbe00700 1 -- 172.21.15.94:0/3867653115 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f48d4020070 con 0x7f48e410ad90 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.275+0000 7f48dbe00700 1 -- 172.21.15.94:0/3867653115 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48d4019630 con 0x7f48e410ad90 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.275+0000 7f48e9000700 1 -- 172.21.15.94:0/3867653115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e419f490 con 0x7f48e410ad90 2024-04-26T19:59:01.974 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.275+0000 7f48e9000700 1 -- 172.21.15.94:0/3867653115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e41a2180 con 0x7f48e410ad90 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.275+0000 7f48dbe00700 1 -- 172.21.15.94:0/3867653115 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f48d4019c80 con 0x7f48e410ad90 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.275+0000 7f48dbe00700 1 -- 172.21.15.94:0/3867653115 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f48d400b720 con 0x7f48e410ad90 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.276+0000 7f48e9000700 1 -- 172.21.15.94:0/3867653115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48e410f340 con 0x7f48e410ad90 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.279+0000 7f48dbe00700 1 -- 172.21.15.94:0/3867653115 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f48d40042a0 con 0x7f48e410ad90 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.356+0000 7f48e9000700 1 -- 172.21.15.94:0/3867653115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f48e41a2350 con 0x7f48e410ad90 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.357+0000 7f48dbe00700 1 -- 172.21.15.94:0/3867653115 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f48d401a9c0 con 0x7f48e410ad90 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.358+0000 7f48e9000700 1 -- 172.21.15.94:0/3867653115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 msgr2=0x7f48e419ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.358+0000 7f48e9000700 1 --2- 172.21.15.94:0/3867653115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e419ecc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48d40117b0 tx=0x7f48d4011b70 comp rx=0 tx=0).stop 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.358+0000 7f48e9000700 1 -- 172.21.15.94:0/3867653115 shutdown_connections 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.359+0000 7f48e9000700 1 --2- 172.21.15.94:0/3867653115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48e410ad90 0x7f48e419ecc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.359+0000 7f48e9000700 1 -- 172.21.15.94:0/3867653115 >> 172.21.15.94:0/3867653115 conn(0x7f48e407e170 msgr2=0x7f48e4108460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.359+0000 7f48e9000700 1 -- 172.21.15.94:0/3867653115 shutdown_connections 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:01.359+0000 7f48e9000700 1 -- 172.21.15.94:0/3867653115 wait complete. 2024-04-26T19:59:01.975 INFO:teuthology.orchestra.run.smithi094.stdout:Restarting the monitor... 2024-04-26T19:59:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:02 smithi094 systemd[1]: Stopping Ceph mon.a for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T19:59:02.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a[27678]: 2024-04-26T19:59:02.448+0000 7f64d2a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T19:59:02.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a[27678]: 2024-04-26T19:59:02.448+0000 7f64d2a00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T19:59:02.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:02 smithi094 podman[27926]: 2024-04-26 19:59:02.562631935 +0000 UTC m=+0.241451396 container died 313cdd3a840b870bf8c6714746e4082e77089eb69bb8953c8bd96ea38bee3a10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T19:59:03.153 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:02 smithi094 podman[27926]: 2024-04-26 19:59:02.934878967 +0000 UTC m=+0.613698469 container cleanup 313cdd3a840b870bf8c6714746e4082e77089eb69bb8953c8bd96ea38bee3a10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T19:59:03.153 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:02 smithi094 bash[27926]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a 2024-04-26T19:59:03.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:03 smithi094 podman[27940]: 2024-04-26 19:59:03.152389418 +0000 UTC m=+0.572013202 container remove 313cdd3a840b870bf8c6714746e4082e77089eb69bb8953c8bd96ea38bee3a10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T19:59:03.977 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:03 smithi094 systemd[1]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T19:59:03.978 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:03 smithi094 systemd[1]: Stopped Ceph mon.a for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T19:59:03.978 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:03 smithi094 systemd[1]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.870s CPU time. 2024-04-26T19:59:03.978 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:03 smithi094 systemd[1]: Starting Ceph mon.a for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T19:59:04.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:03 smithi094 podman[28031]: 2024-04-26 19:59:03.881355145 +0000 UTC m=+0.032213655 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:59:04.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:03 smithi094 podman[28031]: 2024-04-26 19:59:03.978368009 +0000 UTC m=+0.129226556 container create 276d9903869a5947568af1c726803b658a8d96c9f563226a2a912c212472f712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T19:59:04.358 INFO:teuthology.orchestra.run.smithi094.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T19:59:04.492 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 podman[28031]: 2024-04-26 19:59:04.239979889 +0000 UTC m=+0.390838443 container init 276d9903869a5947568af1c726803b658a8d96c9f563226a2a912c212472f712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-26T19:59:04.492 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 podman[28031]: 2024-04-26 19:59:04.250453028 +0000 UTC m=+0.401311581 container start 276d9903869a5947568af1c726803b658a8d96c9f563226a2a912c212472f712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-26T19:59:04.492 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T19:59:04.492 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T19:59:04.492 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: pidfile_write: ignore empty --pid-file 2024-04-26T19:59:04.492 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: load: jerasure load: lrc 2024-04-26T19:59:04.492 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: RocksDB version: 7.9.2 2024-04-26T19:59:04.492 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Git sha 0 2024-04-26T19:59:04.492 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: DB SUMMARY 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: DB Session ID: J7E0ZIIK80DR9GLZCNP7 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: CURRENT file: CURRENT 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104110 ; 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.error_if_exists: 0 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.create_if_missing: 0 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.paranoid_checks: 1 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T19:59:04.493 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T19:59:04.494 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.env: 0x56237e0f2cc0 2024-04-26T19:59:04.494 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T19:59:04.494 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.info_log: 0x56239ec81dc0 2024-04-26T19:59:04.494 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T19:59:04.494 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.statistics: (nil) 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.use_fsync: 0 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_log_file_size: 0 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.allow_fallocate: 1 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T19:59:04.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.use_direct_reads: 0 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.db_log_dir: 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.wal_dir: 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T19:59:04.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.write_buffer_manager: 0x56239ec530e0 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.unordered_write: 0 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T19:59:04.497 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.row_cache: None 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.wal_filter: None 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.two_write_queues: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.wal_compression: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.atomic_flush: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.log_readahead_size: 0 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T19:59:04.498 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_background_jobs: 2 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_background_compactions: -1 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_subcompactions: 1 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T19:59:04.499 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_open_files: -1 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_background_flushes: -1 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Compression algorithms supported: 2024-04-26T19:59:04.500 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: kZSTD supported: 0 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: kXpressCompression supported: 0 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: kZlibCompression supported: 1 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: kSnappyCompression supported: 1 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: kLZ4Compression supported: 1 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: kBZip2Compression supported: 0 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.merge_operator: 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_filter: None 2024-04-26T19:59:04.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56239ec81f80) 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: cache_index_and_filter_blocks: 1 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: pin_top_level_index_and_filter: 1 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: index_type: 0 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: data_block_index_type: 0 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: index_shortening: 1 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: checksum: 4 2024-04-26T19:59:04.502 INFO:journalctl@ceph.mon.a.smithi094.stdout: no_block_cache: 0 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache: 0x56239ec851f0 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache_name: BinnedLRUCache 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache_options: 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: capacity : 536870912 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: num_shard_bits : 4 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: strict_capacity_limit : 0 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: high_pri_pool_ratio: 0.000 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_cache_compressed: (nil) 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: persistent_cache: (nil) 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_size: 4096 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_size_deviation: 10 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_restart_interval: 16 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: index_block_restart_interval: 1 2024-04-26T19:59:04.503 INFO:journalctl@ceph.mon.a.smithi094.stdout: metadata_block_size: 4096 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: partition_filters: 0 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: use_delta_encoding: 1 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: filter_policy: bloomfilter 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: whole_key_filtering: 1 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: verify_compression: 0 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: read_amp_bytes_per_bit: 0 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: format_version: 5 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: enable_index_compression: 1 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: block_align: 0 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: max_auto_readahead_size: 262144 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: prepopulate_block_cache: 0 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: initial_auto_readahead_size: 8192 2024-04-26T19:59:04.504 INFO:journalctl@ceph.mon.a.smithi094.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T19:59:04.505 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T19:59:04.505 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression: NoCompression 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.num_levels: 7 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T19:59:04.506 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T19:59:04.507 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T19:59:04.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T19:59:04.509 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.inplace_update_support: 0 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.bloom_locality: 0 2024-04-26T19:59:04.510 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.max_successive_merges: 0 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.ttl: 2592000 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.enable_blob_files: false 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.min_blob_size: 0 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T19:59:04.511 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5dcfdb7d-e1e7-4c8e-ae80-e8f5e08d93e2 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714161544307546, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714161544309371, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99345, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97291, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13927, "raw_average_key_size": 49, "raw_value_size": 89668, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714161544, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5dcfdb7d-e1e7-4c8e-ae80-e8f5e08d93e2", "db_session_id": "J7E0ZIIK80DR9GLZCNP7", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714161544309566, "job": 1, "event": "recovery_finished"} 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T19:59:04.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56239eda2000 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: DB pointer 0x56239ed8e000 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** DB Stats ** 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-26T19:59:04.513 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** Compaction Stats [default] ** 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: L0 2/0 98.88 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 78.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: Sum 2/0 98.88 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 78.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 78.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** Compaction Stats [default] ** 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T19:59:04.514 INFO:journalctl@ceph.mon.a.smithi094.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 78.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: Cumulative compaction: 0.00 GB write, 15.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: Interval compaction: 0.00 GB write, 15.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: Block cache BinnedLRUCache@0x56239ec851f0#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T19:59:04.515 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: starting mon.a rank 0 at public addrs [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] at bind addrs [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mon.a@-1(???) e1 preinit fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mon.a@-1(???).mds e1 new map 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mon.a@-1(???).mds e1 print_map 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout: e1 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout: legacy client fscid: -1 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout: 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout: No filesystems configured 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T19:59:04.516 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 bash[28031]: 276d9903869a5947568af1c726803b658a8d96c9f563226a2a912c212472f712 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 systemd[1]: Started Ceph mon.a for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: monmap epoch 1 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: last_changed 2024-04-26T19:58:53.653581+0000 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: created 2024-04-26T19:58:53.653581+0000 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: min_mon_release 19 (squid) 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: election_strategy: 1 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: fsmap 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T19:59:04.517 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:04 smithi094 ceph-mon[28050]: mgrmap e1: no daemons active 2024-04-26T19:59:06.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:06 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3558131647' entity='client.admin' 2024-04-26T19:59:13.944 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.029+0000 7fc083200700 1 Processor -- start 2024-04-26T19:59:13.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.029+0000 7fc083200700 1 -- start start 2024-04-26T19:59:13.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.029+0000 7fc083200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c109180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:13.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.029+0000 7fc083200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c109750 con 0x7fc07c108d60 2024-04-26T19:59:13.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.030+0000 7fc07be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c109180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:13.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.030+0000 7fc07be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c109180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52380/0 (socket says 172.21.15.94:52380) 2024-04-26T19:59:13.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.030+0000 7fc07be00700 1 -- 172.21.15.94:0/2535226476 learned_addr learned my addr 172.21.15.94:0/2535226476 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:13.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.031+0000 7fc07be00700 1 -- 172.21.15.94:0/2535226476 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c109f60 con 0x7fc07c108d60 2024-04-26T19:59:13.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.031+0000 7fc07be00700 1 --2- 172.21.15.94:0/2535226476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c109180 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0098a0 tx=0x7fc06c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a763a9f4648f18 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:13.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.031+0000 7fc07aa00700 1 -- 172.21.15.94:0/2535226476 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc06c004160 con 0x7fc07c108d60 2024-04-26T19:59:13.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.031+0000 7fc07aa00700 1 -- 172.21.15.94:0/2535226476 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc06c004300 con 0x7fc07c108d60 2024-04-26T19:59:13.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.031+0000 7fc07aa00700 1 -- 172.21.15.94:0/2535226476 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc06c011af0 con 0x7fc07c108d60 2024-04-26T19:59:13.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.032+0000 7fc083200700 1 -- 172.21.15.94:0/2535226476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 msgr2=0x7fc07c109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.032+0000 7fc083200700 1 --2- 172.21.15.94:0/2535226476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c109180 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0098a0 tx=0x7fc06c009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.032+0000 7fc083200700 1 -- 172.21.15.94:0/2535226476 shutdown_connections 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.032+0000 7fc083200700 1 --2- 172.21.15.94:0/2535226476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c109180 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.032+0000 7fc083200700 1 -- 172.21.15.94:0/2535226476 >> 172.21.15.94:0/2535226476 conn(0x7fc07c104320 msgr2=0x7fc07c106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.032+0000 7fc083200700 1 -- 172.21.15.94:0/2535226476 shutdown_connections 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.032+0000 7fc083200700 1 -- 172.21.15.94:0/2535226476 wait complete. 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.033+0000 7fc083200700 1 Processor -- start 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.033+0000 7fc083200700 1 -- start start 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.034+0000 7fc083200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c07eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.034+0000 7fc083200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c10d190 con 0x7fc07c108d60 2024-04-26T19:59:13.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.034+0000 7fc07be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c07eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.034+0000 7fc07be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c07eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52390/0 (socket says 172.21.15.94:52390) 2024-04-26T19:59:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.034+0000 7fc07be00700 1 -- 172.21.15.94:0/3558131647 learned_addr learned my addr 172.21.15.94:0/3558131647 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.035+0000 7fc07be00700 1 -- 172.21.15.94:0/3558131647 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c07f030 con 0x7fc07c108d60 2024-04-26T19:59:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.035+0000 7fc07be00700 1 --2- 172.21.15.94:0/3558131647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c07eaf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0117b0 tx=0x7fc06c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.035+0000 7fc078c00700 1 -- 172.21.15.94:0/3558131647 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc06c003af0 con 0x7fc07c108d60 2024-04-26T19:59:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.035+0000 7fc078c00700 1 -- 172.21.15.94:0/3558131647 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc06c020070 con 0x7fc07c108d60 2024-04-26T19:59:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.035+0000 7fc083200700 1 -- 172.21.15.94:0/3558131647 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc07c07d220 con 0x7fc07c108d60 2024-04-26T19:59:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.035+0000 7fc078c00700 1 -- 172.21.15.94:0/3558131647 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc06c019630 con 0x7fc07c108d60 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.036+0000 7fc083200700 1 -- 172.21.15.94:0/3558131647 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc07c07d760 con 0x7fc07c108d60 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.036+0000 7fc078c00700 1 -- 172.21.15.94:0/3558131647 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc06c019c80 con 0x7fc07c108d60 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.036+0000 7fc078c00700 1 -- 172.21.15.94:0/3558131647 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc06c00b720 con 0x7fc07c108d60 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.037+0000 7fc083200700 1 -- 172.21.15.94:0/3558131647 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc07c10d310 con 0x7fc07c108d60 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.040+0000 7fc078c00700 1 -- 172.21.15.94:0/3558131647 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fc06c0042a0 con 0x7fc07c108d60 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.116+0000 7fc083200700 1 -- 172.21.15.94:0/3558131647 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fc07c07df00 con 0x7fc07c108d60 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.119+0000 7fc078c00700 1 -- 172.21.15.94:0/3558131647 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fc06c01a9c0 con 0x7fc07c108d60 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.121+0000 7fc083200700 1 -- 172.21.15.94:0/3558131647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 msgr2=0x7fc07c07eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.121+0000 7fc083200700 1 --2- 172.21.15.94:0/3558131647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c07eaf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0117b0 tx=0x7fc06c011b70 comp rx=0 tx=0).stop 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.121+0000 7fc083200700 1 -- 172.21.15.94:0/3558131647 shutdown_connections 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.121+0000 7fc083200700 1 --2- 172.21.15.94:0/3558131647 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc07c108d60 0x7fc07c07eaf0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.121+0000 7fc083200700 1 -- 172.21.15.94:0/3558131647 >> 172.21.15.94:0/3558131647 conn(0x7fc07c104320 msgr2=0x7fc07c104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.121+0000 7fc083200700 1 -- 172.21.15.94:0/3558131647 shutdown_connections 2024-04-26T19:59:13.952 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:05.121+0000 7fc083200700 1 -- 172.21.15.94:0/3558131647 wait complete. 2024-04-26T19:59:13.953 INFO:teuthology.orchestra.run.smithi094.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T19:59:13.974 INFO:teuthology.orchestra.run.smithi094.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T19:59:13.974 INFO:teuthology.orchestra.run.smithi094.stdout:Creating mgr... 2024-04-26T19:59:13.976 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T19:59:13.977 INFO:teuthology.orchestra.run.smithi094.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T19:59:14.413 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.x 2024-04-26T19:59:14.413 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Failed to reset failed state of unit ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T19:59:14.670 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-420d0bd6-0407-11ef-bc93-c7b262605968.target.wants/ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-420d0bd6-0407-11ef-bc93-c7b262605968@.service. 2024-04-26T19:59:15.306 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:15 smithi094 podman[28265]: 2024-04-26 19:59:14.978350074 +0000 UTC m=+0.033220721 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T19:59:15.306 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:15 smithi094 podman[28265]: 2024-04-26 19:59:15.304879193 +0000 UTC m=+0.359749850 container init dd64914477745935a123fae9e5fc59f4a263a11de5f5042830ffd2a3b463084c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-26T19:59:15.427 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T19:59:15.427 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-04-26T19:59:15.442 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T19:59:15.442 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-26T19:59:15.442 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-04-26T19:59:15.442 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T19:59:15.459 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T19:59:15.459 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-04-26T19:59:15.474 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T19:59:15.474 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-26T19:59:15.475 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-04-26T19:59:15.475 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T19:59:15.475 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr to start... 2024-04-26T19:59:15.475 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr... 2024-04-26T19:59:15.628 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:15 smithi094 podman[28265]: 2024-04-26 19:59:15.316072462 +0000 UTC m=+0.370943112 container start dd64914477745935a123fae9e5fc59f4a263a11de5f5042830ffd2a3b463084c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T19:59:15.629 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:15 smithi094 bash[28265]: dd64914477745935a123fae9e5fc59f4a263a11de5f5042830ffd2a3b463084c 2024-04-26T19:59:15.629 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:15 smithi094 systemd[1]: Started Ceph mgr.x for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T19:59:15.629 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:15 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:15.536+0000 7f8969ea1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T19:59:15.922 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:15 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:15.785+0000 7f8969ea1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T19:59:15.922 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:15 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:15.919+0000 7f8969ea1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T19:59:16.463 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:16 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/795325324' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T19:59:16.463 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:16 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:16.349+0000 7f8969ea1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T19:59:16.463 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:16 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:16.461+0000 7f8969ea1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T19:59:16.837 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:16.838 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-26T19:59:16.838 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T19:59:16.838 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T19:59:16.839 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T19:59:16.839 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T19:59:16.839 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T19:59:16.839 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:16.840 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T19:59:16.840 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T19:59:16.840 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-04-26T19:59:16.840 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:16.840 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T19:59:16.841 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T19:59:16.841 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:16.841 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-26T19:59:16.841 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T19:59:16.841 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:16.842 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T19:59:16.842 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T19:59:16.842 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:16.842 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T19:59:16.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:16.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T19:59:16.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T19:59:16.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T19:59:16.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T19:59:16.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T19:59:16.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T19:59:16.856 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:16.856 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T19:59:16.857 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:16.858 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T19:58:57.907400+0000", 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.077+0000 7feba1800700 1 Processor -- start 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.077+0000 7feba1800700 1 -- start start 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.078+0000 7feba1800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c07d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.078+0000 7feba1800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c07d830 con 0x7feb9c07ee00 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.078+0000 7feb9aa00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c07d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.078+0000 7feb9aa00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c07d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34488/0 (socket says 172.21.15.94:34488) 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.078+0000 7feb9aa00700 1 -- 172.21.15.94:0/1126493375 learned_addr learned my addr 172.21.15.94:0/1126493375 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.079+0000 7feb9aa00700 1 -- 172.21.15.94:0/1126493375 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9c07d9b0 con 0x7feb9c07ee00 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.079+0000 7feb9aa00700 1 --2- 172.21.15.94:0/1126493375 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c07d260 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0098a0 tx=0x7feb8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d310a92e89671bee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:16.859 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.079+0000 7feb99600700 1 -- 172.21.15.94:0/1126493375 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb8c004160 con 0x7feb9c07ee00 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.079+0000 7feb99600700 1 -- 172.21.15.94:0/1126493375 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7feb8c004300 con 0x7feb9c07ee00 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.079+0000 7feb99600700 1 -- 172.21.15.94:0/1126493375 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb8c011af0 con 0x7feb9c07ee00 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.080+0000 7feba1800700 1 -- 172.21.15.94:0/1126493375 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 msgr2=0x7feb9c07d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.080+0000 7feba1800700 1 --2- 172.21.15.94:0/1126493375 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c07d260 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0098a0 tx=0x7feb8c009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.080+0000 7feba1800700 1 -- 172.21.15.94:0/1126493375 shutdown_connections 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.080+0000 7feba1800700 1 --2- 172.21.15.94:0/1126493375 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c07d260 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.080+0000 7feba1800700 1 -- 172.21.15.94:0/1126493375 >> 172.21.15.94:0/1126493375 conn(0x7feb9c1042f0 msgr2=0x7feb9c106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.080+0000 7feba1800700 1 -- 172.21.15.94:0/1126493375 shutdown_connections 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.080+0000 7feba1800700 1 -- 172.21.15.94:0/1126493375 wait complete. 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.081+0000 7feba1800700 1 Processor -- start 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.082+0000 7feba1800700 1 -- start start 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.082+0000 7feba1800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.082+0000 7feba1800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c10f2e0 con 0x7feb9c07ee00 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.082+0000 7feb9aa00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.082+0000 7feb9aa00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c19ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34502/0 (socket says 172.21.15.94:34502) 2024-04-26T19:59:16.860 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.082+0000 7feb9aa00700 1 -- 172.21.15.94:0/795325324 learned_addr learned my addr 172.21.15.94:0/795325324 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.083+0000 7feb9aa00700 1 -- 172.21.15.94:0/795325324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9c19f390 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.083+0000 7feb9aa00700 1 --2- 172.21.15.94:0/795325324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c19ee50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feb8c011ac0 tx=0x7feb8c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.083+0000 7feb93400700 1 -- 172.21.15.94:0/795325324 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb8c003970 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.083+0000 7feb93400700 1 -- 172.21.15.94:0/795325324 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7feb8c003b10 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.083+0000 7feba1800700 1 -- 172.21.15.94:0/795325324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb9c19f620 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.083+0000 7feb93400700 1 -- 172.21.15.94:0/795325324 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb8c024610 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.083+0000 7feba1800700 1 -- 172.21.15.94:0/795325324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb9c1a2310 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.084+0000 7feb93400700 1 -- 172.21.15.94:0/795325324 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7feb8c024bd0 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.084+0000 7feb93400700 1 -- 172.21.15.94:0/795325324 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feb8c019700 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.085+0000 7feba1800700 1 -- 172.21.15.94:0/795325324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb7c005360 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.087+0000 7feb93400700 1 -- 172.21.15.94:0/795325324 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7feb8c043ec0 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.163+0000 7feba1800700 1 -- 172.21.15.94:0/795325324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7feb7c005920 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.163+0000 7feb93400700 1 -- 172.21.15.94:0/795325324 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7feb8c032030 con 0x7feb9c07ee00 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.164+0000 7feba1800700 1 -- 172.21.15.94:0/795325324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 msgr2=0x7feb9c19ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:16.861 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.164+0000 7feba1800700 1 --2- 172.21.15.94:0/795325324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c19ee50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feb8c011ac0 tx=0x7feb8c011ba0 comp rx=0 tx=0).stop 2024-04-26T19:59:16.862 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.164+0000 7feba1800700 1 -- 172.21.15.94:0/795325324 shutdown_connections 2024-04-26T19:59:16.862 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.164+0000 7feba1800700 1 --2- 172.21.15.94:0/795325324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb9c07ee00 0x7feb9c19ee50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:16.862 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.164+0000 7feba1800700 1 -- 172.21.15.94:0/795325324 >> 172.21.15.94:0/795325324 conn(0x7feb9c1042f0 msgr2=0x7feb9c104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:16.862 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.165+0000 7feba1800700 1 -- 172.21.15.94:0/795325324 shutdown_connections 2024-04-26T19:59:16.862 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:16.165+0000 7feba1800700 1 -- 172.21.15.94:0/795325324 wait complete. 2024-04-26T19:59:16.862 INFO:teuthology.orchestra.run.smithi094.stdout:mgr not available, waiting (1/15)... 2024-04-26T19:59:18.476 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:18 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:18.044+0000 7f8969ea1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T19:59:18.476 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:18 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:18.264+0000 7f8969ea1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T19:59:18.476 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:18 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:18.474+0000 7f8969ea1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T19:59:19.005 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:18 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:18.849+0000 7f8969ea1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T19:59:19.005 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:18 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:18.974+0000 7f8969ea1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T19:59:19.325 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:19 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:19.233+0000 7f8969ea1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T19:59:19.580 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:19 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:19.490+0000 7f8969ea1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T19:59:19.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:19 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3667924262' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T19:59:20.333 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T19:59:20.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T19:59:20.335 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T19:59:20.343 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T19:59:20.344 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T19:58:57.907400+0000", 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.473+0000 7f7575800700 1 Processor -- start 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.473+0000 7f7575800700 1 -- start start 2024-04-26T19:59:20.345 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.473+0000 7f7575800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f7570109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.473+0000 7f7575800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7570109700 con 0x7f7570108d10 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.473+0000 7f756ea00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f7570109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.474+0000 7f756ea00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f7570109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34516/0 (socket says 172.21.15.94:34516) 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.474+0000 7f756ea00700 1 -- 172.21.15.94:0/3584813246 learned_addr learned my addr 172.21.15.94:0/3584813246 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.474+0000 7f756ea00700 1 -- 172.21.15.94:0/3584813246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7570109f10 con 0x7f7570108d10 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.474+0000 7f756ea00700 1 --2- 172.21.15.94:0/3584813246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f7570109130 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f75600098a0 tx=0x7f7560009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53900b265d5948f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f756d600700 1 -- 172.21.15.94:0/3584813246 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7560004160 con 0x7f7570108d10 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f756d600700 1 -- 172.21.15.94:0/3584813246 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7560004300 con 0x7f7570108d10 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f756d600700 1 -- 172.21.15.94:0/3584813246 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7560011af0 con 0x7f7570108d10 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f7575800700 1 -- 172.21.15.94:0/3584813246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 msgr2=0x7f7570109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f7575800700 1 --2- 172.21.15.94:0/3584813246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f7570109130 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f75600098a0 tx=0x7f7560009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f7575800700 1 -- 172.21.15.94:0/3584813246 shutdown_connections 2024-04-26T19:59:20.346 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f7575800700 1 --2- 172.21.15.94:0/3584813246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f7570109130 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f7575800700 1 -- 172.21.15.94:0/3584813246 >> 172.21.15.94:0/3584813246 conn(0x7f75701042f0 msgr2=0x7f7570106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f7575800700 1 -- 172.21.15.94:0/3584813246 shutdown_connections 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.475+0000 7f7575800700 1 -- 172.21.15.94:0/3584813246 wait complete. 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.476+0000 7f7575800700 1 Processor -- start 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.477+0000 7f7575800700 1 -- start start 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.477+0000 7f7575800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f757007eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.477+0000 7f7575800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f757010d2c0 con 0x7f7570108d10 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.477+0000 7f756ea00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f757007eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.477+0000 7f756ea00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f757007eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34526/0 (socket says 172.21.15.94:34526) 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.477+0000 7f756ea00700 1 -- 172.21.15.94:0/3667924262 learned_addr learned my addr 172.21.15.94:0/3667924262 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.477+0000 7f756ea00700 1 -- 172.21.15.94:0/3667924262 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757007f030 con 0x7f7570108d10 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.478+0000 7f756ea00700 1 --2- 172.21.15.94:0/3667924262 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f757007eaf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75600117b0 tx=0x7f7560011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.478+0000 7f7567400700 1 -- 172.21.15.94:0/3667924262 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7560003af0 con 0x7f7570108d10 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.478+0000 7f7567400700 1 -- 172.21.15.94:0/3667924262 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7560020070 con 0x7f7570108d10 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.478+0000 7f7567400700 1 -- 172.21.15.94:0/3667924262 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7560019630 con 0x7f7570108d10 2024-04-26T19:59:20.347 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.478+0000 7f7575800700 1 -- 172.21.15.94:0/3667924262 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f757007d220 con 0x7f7570108d10 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.478+0000 7f7575800700 1 -- 172.21.15.94:0/3667924262 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f757007d760 con 0x7f7570108d10 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.479+0000 7f7567400700 1 -- 172.21.15.94:0/3667924262 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7560019c30 con 0x7f7570108d10 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.479+0000 7f7567400700 1 -- 172.21.15.94:0/3667924262 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f756000b720 con 0x7f7570108d10 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.479+0000 7f7575800700 1 -- 172.21.15.94:0/3667924262 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7570064a80 con 0x7f7570108d10 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.482+0000 7f7567400700 1 -- 172.21.15.94:0/3667924262 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f75600042c0 con 0x7f7570108d10 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.561+0000 7f7575800700 1 -- 172.21.15.94:0/3667924262 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f757007daa0 con 0x7f7570108d10 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.562+0000 7f7567400700 1 -- 172.21.15.94:0/3667924262 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f756001a9c0 con 0x7f7570108d10 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.563+0000 7f7575800700 1 -- 172.21.15.94:0/3667924262 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 msgr2=0x7f757007eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.563+0000 7f7575800700 1 --2- 172.21.15.94:0/3667924262 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f757007eaf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75600117b0 tx=0x7f7560011b70 comp rx=0 tx=0).stop 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.563+0000 7f7575800700 1 -- 172.21.15.94:0/3667924262 shutdown_connections 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.563+0000 7f7575800700 1 --2- 172.21.15.94:0/3667924262 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7570108d10 0x7f757007eaf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.563+0000 7f7575800700 1 -- 172.21.15.94:0/3667924262 >> 172.21.15.94:0/3667924262 conn(0x7f75701042f0 msgr2=0x7f7570104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.563+0000 7f7575800700 1 -- 172.21.15.94:0/3667924262 shutdown_connections 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:19.563+0000 7f7575800700 1 -- 172.21.15.94:0/3667924262 wait complete. 2024-04-26T19:59:20.348 INFO:teuthology.orchestra.run.smithi094.stdout:mgr not available, waiting (2/15)... 2024-04-26T19:59:20.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:20 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:20.288+0000 7f8969ea1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T19:59:21.220 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:20 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:20.828+0000 7f8969ea1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T19:59:21.220 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:20 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:20.955+0000 7f8969ea1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T19:59:21.220 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:21 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:21.217+0000 7f8969ea1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T19:59:22.237 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:21 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:21.873+0000 7f8969ea1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T19:59:22.237 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:21 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:21.993+0000 7f8969ea1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T19:59:22.630 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:22 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:22.311+0000 7f8969ea1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T19:59:22.631 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:22 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:22.495+0000 7f8969ea1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T19:59:22.897 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:22 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:22.629+0000 7f8969ea1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T19:59:22.898 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:22 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:22.786+0000 7f8969ea1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T19:59:23.165 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2830393014' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T19:59:23.418 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:23 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:23.163+0000 7f8969ea1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T19:59:23.705 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T19:59:23.706 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T19:59:23.707 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T19:59:23.715 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:23.716 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T19:58:57.907400+0000", 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.896+0000 7f833b000700 1 Processor -- start 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.896+0000 7f833b000700 1 -- start start 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.896+0000 7f833b000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833407d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.896+0000 7f833b000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f833407d830 con 0x7f833407ee00 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.897+0000 7f8333e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833407d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.897+0000 7f8333e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833407d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52576/0 (socket says 172.21.15.94:52576) 2024-04-26T19:59:23.717 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.897+0000 7f8333e00700 1 -- 172.21.15.94:0/2432688230 learned_addr learned my addr 172.21.15.94:0/2432688230 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.897+0000 7f8333e00700 1 -- 172.21.15.94:0/2432688230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833407d9b0 con 0x7f833407ee00 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.898+0000 7f8333e00700 1 --2- 172.21.15.94:0/2432688230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833407d260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f83240098a0 tx=0x7f8324009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab4628f269666a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.898+0000 7f8332a00700 1 -- 172.21.15.94:0/2432688230 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8324004160 con 0x7f833407ee00 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.898+0000 7f8332a00700 1 -- 172.21.15.94:0/2432688230 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8324004300 con 0x7f833407ee00 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.898+0000 7f8332a00700 1 -- 172.21.15.94:0/2432688230 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8324011af0 con 0x7f833407ee00 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.899+0000 7f833b000700 1 -- 172.21.15.94:0/2432688230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 msgr2=0x7f833407d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.899+0000 7f833b000700 1 --2- 172.21.15.94:0/2432688230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833407d260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f83240098a0 tx=0x7f8324009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.899+0000 7f833b000700 1 -- 172.21.15.94:0/2432688230 shutdown_connections 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.899+0000 7f833b000700 1 --2- 172.21.15.94:0/2432688230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833407d260 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.899+0000 7f833b000700 1 -- 172.21.15.94:0/2432688230 >> 172.21.15.94:0/2432688230 conn(0x7f83341042f0 msgr2=0x7f8334106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.899+0000 7f833b000700 1 -- 172.21.15.94:0/2432688230 shutdown_connections 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.899+0000 7f833b000700 1 -- 172.21.15.94:0/2432688230 wait complete. 2024-04-26T19:59:23.718 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.900+0000 7f833b000700 1 Processor -- start 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.901+0000 7f833b000700 1 -- start start 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.901+0000 7f833b000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.901+0000 7f833b000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f833410f2e0 con 0x7f833407ee00 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.901+0000 7f8333e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.901+0000 7f8333e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833419ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52592/0 (socket says 172.21.15.94:52592) 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.901+0000 7f8333e00700 1 -- 172.21.15.94:0/2830393014 learned_addr learned my addr 172.21.15.94:0/2830393014 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.902+0000 7f8333e00700 1 -- 172.21.15.94:0/2830393014 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833419f390 con 0x7f833407ee00 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.902+0000 7f8333e00700 1 --2- 172.21.15.94:0/2830393014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833419ee50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8324011ac0 tx=0x7f8324011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.902+0000 7f8330c00700 1 -- 172.21.15.94:0/2830393014 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8324003970 con 0x7f833407ee00 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.902+0000 7f8330c00700 1 -- 172.21.15.94:0/2830393014 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8324003b10 con 0x7f833407ee00 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.902+0000 7f8330c00700 1 -- 172.21.15.94:0/2830393014 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8324024610 con 0x7f833407ee00 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.902+0000 7f833b000700 1 -- 172.21.15.94:0/2830393014 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f833419f620 con 0x7f833407ee00 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.903+0000 7f833b000700 1 -- 172.21.15.94:0/2830393014 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83341a2310 con 0x7f833407ee00 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.903+0000 7f8330c00700 1 -- 172.21.15.94:0/2830393014 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8324024bd0 con 0x7f833407ee00 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.903+0000 7f8330c00700 1 -- 172.21.15.94:0/2830393014 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8324019700 con 0x7f833407ee00 2024-04-26T19:59:23.719 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.904+0000 7f833b000700 1 -- 172.21.15.94:0/2830393014 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8334064a80 con 0x7f833407ee00 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.906+0000 7f8330c00700 1 -- 172.21.15.94:0/2830393014 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f832401a9e0 con 0x7f833407ee00 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.979+0000 7f833b000700 1 -- 172.21.15.94:0/2830393014 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f83341a2860 con 0x7f833407ee00 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.980+0000 7f8330c00700 1 -- 172.21.15.94:0/2830393014 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f832401a430 con 0x7f833407ee00 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.982+0000 7f833b000700 1 -- 172.21.15.94:0/2830393014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 msgr2=0x7f833419ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.982+0000 7f833b000700 1 --2- 172.21.15.94:0/2830393014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833419ee50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8324011ac0 tx=0x7f8324011ba0 comp rx=0 tx=0).stop 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.982+0000 7f833b000700 1 -- 172.21.15.94:0/2830393014 shutdown_connections 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.982+0000 7f833b000700 1 --2- 172.21.15.94:0/2830393014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f833407ee00 0x7f833419ee50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.982+0000 7f833b000700 1 -- 172.21.15.94:0/2830393014 >> 172.21.15.94:0/2830393014 conn(0x7f83341042f0 msgr2=0x7f8334104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.982+0000 7f833b000700 1 -- 172.21.15.94:0/2830393014 shutdown_connections 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:22.982+0000 7f833b000700 1 -- 172.21.15.94:0/2830393014 wait complete. 2024-04-26T19:59:23.720 INFO:teuthology.orchestra.run.smithi094.stdout:mgr not available, waiting (3/15)... 2024-04-26T19:59:23.987 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:23 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:23.917+0000 7f8969ea1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T19:59:24.487 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:24.190+0000 7f8969ea1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: Activating manager daemon x 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: mgrmap e2: x(active, starting, since 0.0109923s) 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: Manager daemon x is now available 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' 2024-04-26T19:59:24.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T19:59:24.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' 2024-04-26T19:59:24.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:24 smithi094 ceph-mon[28050]: from='mgr.14100 172.21.15.94:0/1730453266' entity='mgr.x' 2024-04-26T19:59:26.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:26 smithi094 ceph-mon[28050]: mgrmap e3: x(active, since 1.02175s) 2024-04-26T19:59:27.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:27 smithi094 ceph-mon[28050]: mgrmap e4: x(active, since 2s) 2024-04-26T19:59:27.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:27 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3147121172' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T19:59:27.367 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:27.368 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-26T19:59:27.368 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T19:59:27.368 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T19:59:27.368 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T19:59:27.368 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T19:59:27.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T19:59:27.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:27.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T19:59:27.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T19:59:27.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 0 2024-04-26T19:59:27.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "quorum_age": 22, 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T19:59:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T19:59:27.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T19:59:27.377 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:27.378 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ], 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T19:59:27.379 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T19:58:57.907400+0000", 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout }, 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.330+0000 7f15cd600700 1 Processor -- start 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.330+0000 7f15cd600700 1 -- start start 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.331+0000 7f15cd600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c810bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.331+0000 7f15cd600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f15c807cdd0 con 0x7f15c81097f0 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.331+0000 7f15c6a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c810bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.331+0000 7f15c6a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c810bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52678/0 (socket says 172.21.15.94:52678) 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.331+0000 7f15c6a00700 1 -- 172.21.15.94:0/3912031150 learned_addr learned my addr 172.21.15.94:0/3912031150 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.331+0000 7f15c6a00700 1 -- 172.21.15.94:0/3912031150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c810c120 con 0x7f15c81097f0 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.332+0000 7f15c6a00700 1 --2- 172.21.15.94:0/3912031150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c810bbe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f15b80098a0 tx=0x7f15b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94bd28ffcaad8d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:27.380 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.332+0000 7f15c5600700 1 -- 172.21.15.94:0/3912031150 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15b8004160 con 0x7f15c81097f0 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.332+0000 7f15c5600700 1 -- 172.21.15.94:0/3912031150 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f15b8004300 con 0x7f15c81097f0 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.332+0000 7f15c5600700 1 -- 172.21.15.94:0/3912031150 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15b8011af0 con 0x7f15c81097f0 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.333+0000 7f15cd600700 1 -- 172.21.15.94:0/3912031150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 msgr2=0x7f15c810bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.333+0000 7f15cd600700 1 --2- 172.21.15.94:0/3912031150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c810bbe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f15b80098a0 tx=0x7f15b8009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.333+0000 7f15cd600700 1 -- 172.21.15.94:0/3912031150 shutdown_connections 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.333+0000 7f15cd600700 1 --2- 172.21.15.94:0/3912031150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c810bbe0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.333+0000 7f15cd600700 1 -- 172.21.15.94:0/3912031150 >> 172.21.15.94:0/3912031150 conn(0x7f15c81031e0 msgr2=0x7f15c8105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.333+0000 7f15cd600700 1 -- 172.21.15.94:0/3912031150 shutdown_connections 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.333+0000 7f15cd600700 1 -- 172.21.15.94:0/3912031150 wait complete. 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.334+0000 7f15cd600700 1 Processor -- start 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.334+0000 7f15cd600700 1 -- start start 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.334+0000 7f15cd600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c8109070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.334+0000 7f15cd600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f15c810d090 con 0x7f15c81097f0 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.335+0000 7f15c6a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c8109070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:27.381 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.335+0000 7f15c6a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c8109070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52684/0 (socket says 172.21.15.94:52684) 2024-04-26T19:59:27.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.335+0000 7f15c6a00700 1 -- 172.21.15.94:0/3147121172 learned_addr learned my addr 172.21.15.94:0/3147121172 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:27.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.335+0000 7f15c6a00700 1 -- 172.21.15.94:0/3147121172 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c81095b0 con 0x7f15c81097f0 2024-04-26T19:59:27.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.335+0000 7f15c6a00700 1 --2- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c8109070 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f15b8011ac0 tx=0x7f15b8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:27.384 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.336+0000 7f15bf400700 1 -- 172.21.15.94:0/3147121172 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15b8003970 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.336+0000 7f15bf400700 1 -- 172.21.15.94:0/3147121172 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f15b8003b10 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.336+0000 7f15bf400700 1 -- 172.21.15.94:0/3147121172 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15b8024610 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.336+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c81077a0 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.336+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c8107ce0 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.337+0000 7f15bf400700 1 -- 172.21.15.94:0/3147121172 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f15b801a430 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.337+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15c8064a80 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.337+0000 7f15bf400700 1 --2- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f15ac0395b0 0x7f15ac03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.337+0000 7f15bf400700 1 -- 172.21.15.94:0/3147121172 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f15b804d850 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.338+0000 7f15c6000700 1 --2- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f15ac0395b0 0x7f15ac03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.338+0000 7f15c6000700 1 --2- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f15ac0395b0 0x7f15ac03ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f15b4006fd0 tx=0x7f15b4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.344+0000 7f15bf400700 1 -- 172.21.15.94:0/3147121172 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f15b8016260 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.654+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f15c8107f60 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.654+0000 7f15bf400700 1 -- 172.21.15.94:0/3147121172 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f15b804d270 con 0x7f15c81097f0 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.657+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f15ac0395b0 msgr2=0x7f15ac03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.657+0000 7f15cd600700 1 --2- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f15ac0395b0 0x7f15ac03ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f15b4006fd0 tx=0x7f15b4006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:27.385 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.657+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 msgr2=0x7f15c8109070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.657+0000 7f15cd600700 1 --2- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c8109070 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f15b8011ac0 tx=0x7f15b8011ba0 comp rx=0 tx=0).stop 2024-04-26T19:59:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.657+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 shutdown_connections 2024-04-26T19:59:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.657+0000 7f15cd600700 1 --2- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f15ac0395b0 0x7f15ac03ba70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.657+0000 7f15cd600700 1 --2- 172.21.15.94:0/3147121172 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c81097f0 0x7f15c8109070 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.657+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 >> 172.21.15.94:0/3147121172 conn(0x7f15c81031e0 msgr2=0x7f15c8103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.658+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 shutdown_connections 2024-04-26T19:59:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:26.658+0000 7f15cd600700 1 -- 172.21.15.94:0/3147121172 wait complete. 2024-04-26T19:59:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:mgr is available 2024-04-26T19:59:28.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:28 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2610982001' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T19:59:28.893 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:28.893 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T19:59:28.893 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout fsid = 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:59:28.893 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T19:59:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T19:59:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T19:59:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T19:59:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T19:59:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T19:59:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T19:59:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 2024-04-26T19:59:28.895 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T19:59:28.896 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T19:59:28.896 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T19:59:28.896 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T19:59:28.896 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T19:59:28.899 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T19:59:28.899 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.948+0000 7f84d5e00700 1 Processor -- start 2024-04-26T19:59:28.899 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.948+0000 7f84d5e00700 1 -- start start 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.948+0000 7f84d5e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d0112040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.948+0000 7f84d5e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f84d00833f0 con 0x7f84d010fc50 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.949+0000 7f84d4a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d0112040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.949+0000 7f84d4a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d0112040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52692/0 (socket says 172.21.15.94:52692) 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.949+0000 7f84d4a00700 1 -- 172.21.15.94:0/710532969 learned_addr learned my addr 172.21.15.94:0/710532969 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.949+0000 7f84d4a00700 1 -- 172.21.15.94:0/710532969 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d0112580 con 0x7f84d010fc50 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.950+0000 7f84d4a00700 1 --2- 172.21.15.94:0/710532969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d0112040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f84c40098a0 tx=0x7f84c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5d146374578011 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.950+0000 7f84cf400700 1 -- 172.21.15.94:0/710532969 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c4004160 con 0x7f84d010fc50 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.950+0000 7f84cf400700 1 -- 172.21.15.94:0/710532969 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f84c4004300 con 0x7f84d010fc50 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.950+0000 7f84cf400700 1 -- 172.21.15.94:0/710532969 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c4011af0 con 0x7f84d010fc50 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.950+0000 7f84d5e00700 1 -- 172.21.15.94:0/710532969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 msgr2=0x7f84d0112040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.950+0000 7f84d5e00700 1 --2- 172.21.15.94:0/710532969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d0112040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f84c40098a0 tx=0x7f84c4009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.951+0000 7f84d5e00700 1 -- 172.21.15.94:0/710532969 shutdown_connections 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.951+0000 7f84d5e00700 1 --2- 172.21.15.94:0/710532969 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d0112040 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:28.900 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.951+0000 7f84d5e00700 1 -- 172.21.15.94:0/710532969 >> 172.21.15.94:0/710532969 conn(0x7f84d007e170 msgr2=0x7f84d0080590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.951+0000 7f84d5e00700 1 -- 172.21.15.94:0/710532969 shutdown_connections 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.951+0000 7f84d5e00700 1 -- 172.21.15.94:0/710532969 wait complete. 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.952+0000 7f84d5e00700 1 Processor -- start 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.952+0000 7f84d5e00700 1 -- start start 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.952+0000 7f84d5e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d01a3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.952+0000 7f84d5e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f84d0113480 con 0x7f84d010fc50 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.952+0000 7f84d4a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d01a3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.953+0000 7f84d4a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d01a3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52700/0 (socket says 172.21.15.94:52700) 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.953+0000 7f84d4a00700 1 -- 172.21.15.94:0/2610982001 learned_addr learned my addr 172.21.15.94:0/2610982001 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.953+0000 7f84d4a00700 1 -- 172.21.15.94:0/2610982001 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d01a35c0 con 0x7f84d010fc50 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.953+0000 7f84d4a00700 1 --2- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d01a3080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84c4011ac0 tx=0x7f84c4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.953+0000 7f84cd600700 1 -- 172.21.15.94:0/2610982001 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c4003970 con 0x7f84d010fc50 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.954+0000 7f84cd600700 1 -- 172.21.15.94:0/2610982001 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f84c4003b10 con 0x7f84d010fc50 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.954+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d01a3850 con 0x7f84d010fc50 2024-04-26T19:59:28.901 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.954+0000 7f84cd600700 1 -- 172.21.15.94:0/2610982001 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c4024610 con 0x7f84d010fc50 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.954+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d01a6540 con 0x7f84d010fc50 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.955+0000 7f84cd600700 1 -- 172.21.15.94:0/2610982001 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f84c401a9e0 con 0x7f84d010fc50 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.955+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d0064a80 con 0x7f84d010fc50 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.955+0000 7f84cd600700 1 --2- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f84b80395b0 0x7f84b803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.955+0000 7f84cd600700 1 -- 172.21.15.94:0/2610982001 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f84c404d860 con 0x7f84d010fc50 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.956+0000 7f84cfe00700 1 --2- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f84b80395b0 0x7f84b803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.956+0000 7f84cfe00700 1 --2- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f84b80395b0 0x7f84b803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f84bc006fd0 tx=0x7f84bc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:27.960+0000 7f84cd600700 1 -- 172.21.15.94:0/2610982001 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f84c400b3e0 con 0x7f84d010fc50 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.183+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f84d01a6710 con 0x7f84d010fc50 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.184+0000 7f84cd600700 1 -- 172.21.15.94:0/2610982001 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f84d01a6710 con 0x7f84d010fc50 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f84b80395b0 msgr2=0x7f84b803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:28.904 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 --2- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f84b80395b0 0x7f84b803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f84bc006fd0 tx=0x7f84bc006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:28.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 msgr2=0x7f84d01a3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:28.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 --2- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d01a3080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84c4011ac0 tx=0x7f84c4011ba0 comp rx=0 tx=0).stop 2024-04-26T19:59:28.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 shutdown_connections 2024-04-26T19:59:28.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 --2- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f84b80395b0 0x7f84b803ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:28.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 --2- 172.21.15.94:0/2610982001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84d010fc50 0x7f84d01a3080 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:28.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 >> 172.21.15.94:0/2610982001 conn(0x7f84d007e170 msgr2=0x7f84d007ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:28.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 shutdown_connections 2024-04-26T19:59:28.905 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:28.188+0000 7f84d5e00700 1 -- 172.21.15.94:0/2610982001 wait complete. 2024-04-26T19:59:28.905 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling cephadm module... 2024-04-26T19:59:30.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:29 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/787163264' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T19:59:30.219 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:30 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: ignoring --setuser ceph since I am not root 2024-04-26T19:59:30.219 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:30 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: ignoring --setgroup ceph since I am not root 2024-04-26T19:59:30.474 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:30 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:30.216+0000 7fc511b45200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T19:59:30.474 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:30 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:30.472+0000 7fc511b45200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T19:59:30.650 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.565+0000 7f2161200700 1 Processor -- start 2024-04-26T19:59:30.650 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.565+0000 7f2161200700 1 -- start start 2024-04-26T19:59:30.650 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.566+0000 7f2161200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c10b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:30.650 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.566+0000 7f2161200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f215c10b8c0 con 0x7f215c10af60 2024-04-26T19:59:30.650 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.566+0000 7f2159e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c10b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:30.650 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.566+0000 7f2159e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c10b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52704/0 (socket says 172.21.15.94:52704) 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.566+0000 7f2159e00700 1 -- 172.21.15.94:0/1021463165 learned_addr learned my addr 172.21.15.94:0/1021463165 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.567+0000 7f2159e00700 1 -- 172.21.15.94:0/1021463165 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215c10c070 con 0x7f215c10af60 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.567+0000 7f2159e00700 1 --2- 172.21.15.94:0/1021463165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c10b380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f214c0098a0 tx=0x7f214c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b9abe978a7a4ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.567+0000 7f2158a00700 1 -- 172.21.15.94:0/1021463165 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f214c004160 con 0x7f215c10af60 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.568+0000 7f2158a00700 1 -- 172.21.15.94:0/1021463165 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f214c004300 con 0x7f215c10af60 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.568+0000 7f2158a00700 1 -- 172.21.15.94:0/1021463165 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f214c011af0 con 0x7f215c10af60 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.568+0000 7f2161200700 1 -- 172.21.15.94:0/1021463165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 msgr2=0x7f215c10b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.568+0000 7f2161200700 1 --2- 172.21.15.94:0/1021463165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c10b380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f214c0098a0 tx=0x7f214c009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.568+0000 7f2161200700 1 -- 172.21.15.94:0/1021463165 shutdown_connections 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.568+0000 7f2161200700 1 --2- 172.21.15.94:0/1021463165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c10b380 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:30.651 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.568+0000 7f2161200700 1 -- 172.21.15.94:0/1021463165 >> 172.21.15.94:0/1021463165 conn(0x7f215c07e320 msgr2=0x7f215c07e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.569+0000 7f2161200700 1 -- 172.21.15.94:0/1021463165 shutdown_connections 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.569+0000 7f2161200700 1 -- 172.21.15.94:0/1021463165 wait complete. 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.570+0000 7f2161200700 1 Processor -- start 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.570+0000 7f2161200700 1 -- start start 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.570+0000 7f2161200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.570+0000 7f2161200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f215c10f4b0 con 0x7f215c10af60 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.570+0000 7f2159e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.570+0000 7f2159e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c19ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52708/0 (socket says 172.21.15.94:52708) 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.570+0000 7f2159e00700 1 -- 172.21.15.94:0/787163264 learned_addr learned my addr 172.21.15.94:0/787163264 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.571+0000 7f2159e00700 1 -- 172.21.15.94:0/787163264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215c19f3a0 con 0x7f215c10af60 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.571+0000 7f2159e00700 1 --2- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c19ee60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f214c011ac0 tx=0x7f214c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:30.652 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.571+0000 7f2152a00700 1 -- 172.21.15.94:0/787163264 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f214c003970 con 0x7f215c10af60 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.571+0000 7f2152a00700 1 -- 172.21.15.94:0/787163264 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f214c003b10 con 0x7f215c10af60 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.571+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215c19f630 con 0x7f215c10af60 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.571+0000 7f2152a00700 1 -- 172.21.15.94:0/787163264 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f214c024610 con 0x7f215c10af60 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.571+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f215c1a2320 con 0x7f215c10af60 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.573+0000 7f2152a00700 1 -- 172.21.15.94:0/787163264 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f214c01a9e0 con 0x7f215c10af60 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.573+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f215c064a80 con 0x7f215c10af60 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.573+0000 7f2152a00700 1 --2- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f21400395b0 0x7f214003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.573+0000 7f2152a00700 1 -- 172.21.15.94:0/787163264 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f214c04d880 con 0x7f215c10af60 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.574+0000 7f2159400700 1 --2- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f21400395b0 0x7f214003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.574+0000 7f2159400700 1 --2- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f21400395b0 0x7f214003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2148006fd0 tx=0x7f2148006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:30.653 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.579+0000 7f2152a00700 1 -- 172.21.15.94:0/787163264 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f214c019dd0 con 0x7f215c10af60 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.868+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f215c1a24f0 con 0x7f215c10af60 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.921+0000 7f2152a00700 1 -- 172.21.15.94:0/787163264 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f215c1a24f0 con 0x7f215c10af60 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.925+0000 7f2152a00700 1 -- 172.21.15.94:0/787163264 <== mon.0 v2:172.21.15.94:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f214c04fcc0 con 0x7f215c10af60 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f21400395b0 msgr2=0x7f214003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 --2- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f21400395b0 0x7f214003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2148006fd0 tx=0x7f2148006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 msgr2=0x7f215c19ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 --2- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c19ee60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f214c011ac0 tx=0x7f214c011ba0 comp rx=0 tx=0).stop 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 shutdown_connections 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 --2- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f21400395b0 0x7f214003ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 --2- 172.21.15.94:0/787163264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215c10af60 0x7f215c19ee60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:30.654 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 >> 172.21.15.94:0/787163264 conn(0x7f215c07e320 msgr2=0x7f215c1083d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:30.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 shutdown_connections 2024-04-26T19:59:30.655 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:29.926+0000 7f2161200700 1 -- 172.21.15.94:0/787163264 wait complete. 2024-04-26T19:59:30.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:30 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:30.609+0000 7fc511b45200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T19:59:31.121 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:30 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/787163264' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T19:59:31.142 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:30 smithi094 ceph-mon[28050]: mgrmap e5: x(active, since 5s) 2024-04-26T19:59:31.142 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:31 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:31.007+0000 7fc511b45200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T19:59:31.486 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:31 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:31.118+0000 7fc511b45200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T19:59:32.231 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:31 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3737418002' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T19:59:32.242 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-26T19:59:32.242 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T19:59:32.242 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.276+0000 7f8b70e00700 1 Processor -- start 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.276+0000 7f8b70e00700 1 -- start start 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.277+0000 7f8b70e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c07d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.277+0000 7f8b70e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c07d830 con 0x7f8b6c07ee00 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.277+0000 7f8b69e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c07d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.277+0000 7f8b69e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c07d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47352/0 (socket says 172.21.15.94:47352) 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.277+0000 7f8b69e00700 1 -- 172.21.15.94:0/3266855662 learned_addr learned my addr 172.21.15.94:0/3266855662 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.278+0000 7f8b69e00700 1 -- 172.21.15.94:0/3266855662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6c07d9b0 con 0x7f8b6c07ee00 2024-04-26T19:59:32.243 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.278+0000 7f8b69e00700 1 --2- 172.21.15.94:0/3266855662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c07d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c0098a0 tx=0x7f8b5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648d541062e02c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:32.247 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.278+0000 7f8b68a00700 1 -- 172.21.15.94:0/3266855662 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b5c004160 con 0x7f8b6c07ee00 2024-04-26T19:59:32.247 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.278+0000 7f8b68a00700 1 -- 172.21.15.94:0/3266855662 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8b5c004300 con 0x7f8b6c07ee00 2024-04-26T19:59:32.247 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.278+0000 7f8b68a00700 1 -- 172.21.15.94:0/3266855662 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b5c011af0 con 0x7f8b6c07ee00 2024-04-26T19:59:32.247 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.279+0000 7f8b70e00700 1 -- 172.21.15.94:0/3266855662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 msgr2=0x7f8b6c07d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:32.247 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.279+0000 7f8b70e00700 1 --2- 172.21.15.94:0/3266855662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c07d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c0098a0 tx=0x7f8b5c009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:32.247 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.279+0000 7f8b70e00700 1 -- 172.21.15.94:0/3266855662 shutdown_connections 2024-04-26T19:59:32.247 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.279+0000 7f8b70e00700 1 --2- 172.21.15.94:0/3266855662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c07d260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:32.247 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.279+0000 7f8b70e00700 1 -- 172.21.15.94:0/3266855662 >> 172.21.15.94:0/3266855662 conn(0x7f8b6c1042a0 msgr2=0x7f8b6c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.279+0000 7f8b70e00700 1 -- 172.21.15.94:0/3266855662 shutdown_connections 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.279+0000 7f8b70e00700 1 -- 172.21.15.94:0/3266855662 wait complete. 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.280+0000 7f8b70e00700 1 Processor -- start 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.280+0000 7f8b70e00700 1 -- start start 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.280+0000 7f8b70e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.281+0000 7f8b70e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c10f2b0 con 0x7f8b6c07ee00 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.281+0000 7f8b69e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.281+0000 7f8b69e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c19ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47364/0 (socket says 172.21.15.94:47364) 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.281+0000 7f8b69e00700 1 -- 172.21.15.94:0/3737418002 learned_addr learned my addr 172.21.15.94:0/3737418002 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.281+0000 7f8b69e00700 1 -- 172.21.15.94:0/3737418002 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6c19f380 con 0x7f8b6c07ee00 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.281+0000 7f8b69e00700 1 --2- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c19ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c011ac0 tx=0x7f8b5c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.282+0000 7f8b62a00700 1 -- 172.21.15.94:0/3737418002 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b5c003970 con 0x7f8b6c07ee00 2024-04-26T19:59:32.248 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.282+0000 7f8b62a00700 1 -- 172.21.15.94:0/3737418002 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8b5c003b10 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.282+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b6c19f610 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.282+0000 7f8b62a00700 1 -- 172.21.15.94:0/3737418002 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b5c024610 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.282+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b6c1a2300 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.283+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b6c064a80 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.283+0000 7f8b62a00700 1 -- 172.21.15.94:0/3737418002 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f8b5c024840 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.284+0000 7f8b62a00700 1 --2- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f8b50039660 0x7f8b5003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.284+0000 7f8b62a00700 1 -- 172.21.15.94:0/3737418002 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8b5c04d8d0 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.284+0000 7f8b69400700 1 -- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f8b50039660 msgr2=0x7f8b5003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3005317891 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.284+0000 7f8b69400700 1 --2- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f8b50039660 0x7f8b5003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.289+0000 7f8b62a00700 1 -- 172.21.15.94:0/3737418002 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8b5c0172b0 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.485+0000 7f8b69400700 1 -- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f8b50039660 msgr2=0x7f8b5003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3005317891 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.485+0000 7f8b69400700 1 --2- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f8b50039660 0x7f8b5003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.556+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f8b6c1a24d0 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.557+0000 7f8b62a00700 1 -- 172.21.15.94:0/3737418002 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f8b6c1a24d0 con 0x7f8b6c07ee00 2024-04-26T19:59:32.249 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.561+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f8b50039660 msgr2=0x7f8b5003bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.561+0000 7f8b70e00700 1 --2- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f8b50039660 0x7f8b5003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.561+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 msgr2=0x7f8b6c19ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.561+0000 7f8b70e00700 1 --2- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c19ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c011ac0 tx=0x7f8b5c011ba0 comp rx=0 tx=0).stop 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.561+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 shutdown_connections 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.561+0000 7f8b70e00700 1 --2- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f8b50039660 0x7f8b5003bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.561+0000 7f8b70e00700 1 --2- 172.21.15.94:0/3737418002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b6c07ee00 0x7f8b6c19ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.561+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 >> 172.21.15.94:0/3737418002 conn(0x7f8b6c1042a0 msgr2=0x7f8b6c104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.562+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 shutdown_connections 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:31.562+0000 7f8b70e00700 1 -- 172.21.15.94:0/3737418002 wait complete. 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for the mgr to restart... 2024-04-26T19:59:32.250 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr epoch 5... 2024-04-26T19:59:32.797 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:32 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:32.669+0000 7fc511b45200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T19:59:33.110 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:32 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:32.899+0000 7fc511b45200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T19:59:33.110 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:33 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:33.108+0000 7fc511b45200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T19:59:33.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:33 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:33.477+0000 7fc511b45200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T19:59:33.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:33 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:33.596+0000 7fc511b45200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T19:59:34.237 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:33 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:33.845+0000 7fc511b45200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T19:59:34.237 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:34 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:34.091+0000 7fc511b45200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T19:59:35.237 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:34 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:34.864+0000 7fc511b45200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T19:59:35.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:35 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:35.401+0000 7fc511b45200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T19:59:35.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:35 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:35.526+0000 7fc511b45200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T19:59:36.237 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:35 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:35.784+0000 7fc511b45200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T19:59:36.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:36 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:36.419+0000 7fc511b45200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T19:59:36.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:36 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:36.542+0000 7fc511b45200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T19:59:37.168 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:36 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:36.861+0000 7fc511b45200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T19:59:37.168 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:37 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:37.037+0000 7fc511b45200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T19:59:37.168 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:37 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:37.166+0000 7fc511b45200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T19:59:37.487 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:37 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:37.316+0000 7fc511b45200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T19:59:37.987 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:37 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:37.685+0000 7fc511b45200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T19:59:38.715 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:38.429+0000 7fc511b45200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T19:59:38.987 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T19:59:38.713+0000 7fc511b45200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T19:59:38.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: Active manager daemon x restarted 2024-04-26T19:59:38.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: Activating manager daemon x 2024-04-26T19:59:38.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T19:59:38.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: mgrmap e6: x(active, starting, since 0.00464172s) 2024-04-26T19:59:38.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T19:59:38.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T19:59:38.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T19:59:38.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T19:59:38.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T19:59:38.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: Manager daemon x is now available 2024-04-26T19:59:38.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:38.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:38 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:39.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:39 smithi094 ceph-mon[28050]: Found migration_current of "None". Setting to last migration. 2024-04-26T19:59:39.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:39 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:59:39.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:39 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:59:39.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:39 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T19:59:39.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:39 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T19:59:39.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:39 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:39.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:39 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:39.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:39 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:59:39.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:39 smithi094 ceph-mon[28050]: mgrmap e7: x(active, since 1.01659s) 2024-04-26T19:59:40.480 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-26T19:59:40.480 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T19:59:40.480 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.829+0000 7f9f4e600700 1 Processor -- start 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.829+0000 7f9f4e600700 1 -- start start 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.829+0000 7f9f4e600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4807d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.829+0000 7f9f4e600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4807d870 con 0x7f9f4807ee40 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.830+0000 7f9f47400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4807d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.830+0000 7f9f47400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4807d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47366/0 (socket says 172.21.15.94:47366) 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.830+0000 7f9f47400700 1 -- 172.21.15.94:0/1350542229 learned_addr learned my addr 172.21.15.94:0/1350542229 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.830+0000 7f9f47400700 1 -- 172.21.15.94:0/1350542229 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4807d9f0 con 0x7f9f4807ee40 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.830+0000 7f9f47400700 1 --2- 172.21.15.94:0/1350542229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4807d2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9f380098a0 tx=0x7f9f38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc395724605eb7a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.831+0000 7f9f46000700 1 -- 172.21.15.94:0/1350542229 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f38004160 con 0x7f9f4807ee40 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.831+0000 7f9f46000700 1 -- 172.21.15.94:0/1350542229 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9f38004300 con 0x7f9f4807ee40 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.831+0000 7f9f46000700 1 -- 172.21.15.94:0/1350542229 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f38011af0 con 0x7f9f4807ee40 2024-04-26T19:59:40.481 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.831+0000 7f9f4e600700 1 -- 172.21.15.94:0/1350542229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 msgr2=0x7f9f4807d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.831+0000 7f9f4e600700 1 --2- 172.21.15.94:0/1350542229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4807d2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9f380098a0 tx=0x7f9f38009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.832+0000 7f9f4e600700 1 -- 172.21.15.94:0/1350542229 shutdown_connections 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.832+0000 7f9f4e600700 1 --2- 172.21.15.94:0/1350542229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4807d2a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.832+0000 7f9f4e600700 1 -- 172.21.15.94:0/1350542229 >> 172.21.15.94:0/1350542229 conn(0x7f9f481042f0 msgr2=0x7f9f48106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.832+0000 7f9f4e600700 1 -- 172.21.15.94:0/1350542229 shutdown_connections 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.832+0000 7f9f4e600700 1 -- 172.21.15.94:0/1350542229 wait complete. 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.833+0000 7f9f4e600700 1 Processor -- start 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.833+0000 7f9f4e600700 1 -- start start 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.833+0000 7f9f4e600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.833+0000 7f9f4e600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4810f2a0 con 0x7f9f4807ee40 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.833+0000 7f9f47400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.833+0000 7f9f47400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4819ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47372/0 (socket says 172.21.15.94:47372) 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.833+0000 7f9f47400700 1 -- 172.21.15.94:0/2764388564 learned_addr learned my addr 172.21.15.94:0/2764388564 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:40.482 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.834+0000 7f9f47400700 1 -- 172.21.15.94:0/2764388564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4819f3a0 con 0x7f9f4807ee40 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.834+0000 7f9f47400700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4819ee60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9f380117b0 tx=0x7f9f38011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.834+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f38003af0 con 0x7f9f4807ee40 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.834+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9f38020070 con 0x7f9f4807ee40 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.834+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f38019630 con 0x7f9f4807ee40 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.834+0000 7f9f4e600700 1 -- 172.21.15.94:0/2764388564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f4819f630 con 0x7f9f4807ee40 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.835+0000 7f9f4e600700 1 -- 172.21.15.94:0/2764388564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f481a2320 con 0x7f9f4807ee40 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.835+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f9f380198a0 con 0x7f9f4807ee40 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.836+0000 7f9f3fe00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 0x7f9f2c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.836+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 --> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9f2c03c1d0 con 0x7f9f2c039600 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.836+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9f3804dd20 con 0x7f9f4807ee40 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.836+0000 7f9f46a00700 1 -- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 msgr2=0x7f9f2c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3005317891 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:32.836+0000 7f9f46a00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 0x7f9f2c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T19:59:40.483 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:33.037+0000 7f9f46a00700 1 -- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 msgr2=0x7f9f2c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3005317891 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:33.037+0000 7f9f46a00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 0x7f9f2c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:33.438+0000 7f9f46a00700 1 -- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 msgr2=0x7f9f2c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3005317891 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:33.438+0000 7f9f46a00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 0x7f9f2c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:34.239+0000 7f9f46a00700 1 -- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 msgr2=0x7f9f2c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3005317891 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:34.239+0000 7f9f46a00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 0x7f9f2c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:35.841+0000 7f9f46a00700 1 -- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 msgr2=0x7f9f2c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3005317891 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:35.841+0000 7f9f46a00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 0x7f9f2c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:38.721+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f9f3800b480 con 0x7f9f4807ee40 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:38.721+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 msgr2=0x7f9f2c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:38.721+0000 7f9f3fe00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 0x7f9f2c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.728+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f9f3804f340 con 0x7f9f4807ee40 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.728+0000 7f9f3fe00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9f2c03cef0 0x7f9f2c03f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.728+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9f38001d30 con 0x7f9f2c03cef0 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.730+0000 7f9f46a00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9f2c03cef0 0x7f9f2c03f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:40.484 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.732+0000 7f9f46a00700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9f2c03cef0 0x7f9f2c03f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f34003a10 tx=0x7f9f340092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.734+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 <== mgr.14120 v2:172.21.15.94:6800/3651438916 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f9f38001d30 con 0x7f9f2c03cef0 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.743+0000 7f9f4e600700 1 -- 172.21.15.94:0/2764388564 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9f48064a80 con 0x7f9f2c03cef0 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.744+0000 7f9f3fe00700 1 -- 172.21.15.94:0/2764388564 <== mgr.14120 v2:172.21.15.94:6800/3651438916 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f9f48064a80 con 0x7f9f2c03cef0 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.744+0000 7f9f4e600700 1 -- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9f2c03cef0 msgr2=0x7f9f2c03f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.744+0000 7f9f4e600700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9f2c03cef0 0x7f9f2c03f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f34003a10 tx=0x7f9f340092b0 comp rx=0 tx=0).stop 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.744+0000 7f9f4e600700 1 -- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 msgr2=0x7f9f4819ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.744+0000 7f9f4e600700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4819ee60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9f380117b0 tx=0x7f9f38011b70 comp rx=0 tx=0).stop 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.745+0000 7f9f4e600700 1 -- 172.21.15.94:0/2764388564 shutdown_connections 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.745+0000 7f9f4e600700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9f2c03cef0 0x7f9f2c03f2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.745+0000 7f9f4e600700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f4807ee40 0x7f9f4819ee60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.745+0000 7f9f4e600700 1 --2- 172.21.15.94:0/2764388564 >> [v2:172.21.15.94:6800/3005317891,v1:172.21.15.94:6801/3005317891] conn(0x7f9f2c039600 0x7f9f2c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.745+0000 7f9f4e600700 1 -- 172.21.15.94:0/2764388564 >> 172.21.15.94:0/2764388564 conn(0x7f9f481042f0 msgr2=0x7f9f48104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.745+0000 7f9f4e600700 1 -- 172.21.15.94:0/2764388564 shutdown_connections 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:39.745+0000 7f9f4e600700 1 -- 172.21.15.94:0/2764388564 wait complete. 2024-04-26T19:59:40.485 INFO:teuthology.orchestra.run.smithi094.stdout:mgr epoch 5 is available 2024-04-26T19:59:40.486 INFO:teuthology.orchestra.run.smithi094.stdout:Setting orchestrator backend to cephadm... 2024-04-26T19:59:41.020 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:40 smithi094 ceph-mon[28050]: [26/Apr/2024:19:59:39] ENGINE Bus STARTING 2024-04-26T19:59:41.020 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:40 smithi094 ceph-mon[28050]: [26/Apr/2024:19:59:39] ENGINE Serving on https://172.21.15.94:7150 2024-04-26T19:59:41.020 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:40 smithi094 ceph-mon[28050]: [26/Apr/2024:19:59:39] ENGINE Serving on http://172.21.15.94:8765 2024-04-26T19:59:41.020 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:40 smithi094 ceph-mon[28050]: [26/Apr/2024:19:59:39] ENGINE Bus STARTED 2024-04-26T19:59:41.020 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:40 smithi094 ceph-mon[28050]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T19:59:41.021 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:40 smithi094 ceph-mon[28050]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T19:59:41.975 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:41 smithi094 ceph-mon[28050]: mgrmap e8: x(active, since 2s) 2024-04-26T19:59:41.975 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:41 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:41.975 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:41 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:59:41.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.050+0000 7fda12000700 1 Processor -- start 2024-04-26T19:59:41.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.050+0000 7fda12000700 1 -- start start 2024-04-26T19:59:41.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.051+0000 7fda12000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c07d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:41.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.051+0000 7fda12000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c07d870 con 0x7fda0c07ee40 2024-04-26T19:59:41.996 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.051+0000 7fda0b400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c07d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.051+0000 7fda0b400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c07d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44994/0 (socket says 172.21.15.94:44994) 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.051+0000 7fda0b400700 1 -- 172.21.15.94:0/1126297430 learned_addr learned my addr 172.21.15.94:0/1126297430 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.052+0000 7fda0b400700 1 -- 172.21.15.94:0/1126297430 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0c07d9f0 con 0x7fda0c07ee40 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.052+0000 7fda0b400700 1 --2- 172.21.15.94:0/1126297430 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c07d2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009e80 tx=0x7fd9fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77deb5b39d9487b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.052+0000 7fda0a000700 1 -- 172.21.15.94:0/1126297430 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9fc017070 con 0x7fda0c07ee40 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.053+0000 7fda0a000700 1 -- 172.21.15.94:0/1126297430 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9fc00b8c0 con 0x7fda0c07ee40 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.053+0000 7fda0a000700 1 -- 172.21.15.94:0/1126297430 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9fc004490 con 0x7fda0c07ee40 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.054+0000 7fda12000700 1 -- 172.21.15.94:0/1126297430 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 msgr2=0x7fda0c07d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.054+0000 7fda12000700 1 --2- 172.21.15.94:0/1126297430 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c07d2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009e80 tx=0x7fd9fc00b330 comp rx=0 tx=0).stop 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.054+0000 7fda12000700 1 -- 172.21.15.94:0/1126297430 shutdown_connections 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.054+0000 7fda12000700 1 --2- 172.21.15.94:0/1126297430 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c07d2a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.054+0000 7fda12000700 1 -- 172.21.15.94:0/1126297430 >> 172.21.15.94:0/1126297430 conn(0x7fda0c1042f0 msgr2=0x7fda0c106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.054+0000 7fda12000700 1 -- 172.21.15.94:0/1126297430 shutdown_connections 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.054+0000 7fda12000700 1 -- 172.21.15.94:0/1126297430 wait complete. 2024-04-26T19:59:41.997 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.055+0000 7fda12000700 1 Processor -- start 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.055+0000 7fda12000700 1 -- start start 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.055+0000 7fda12000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.055+0000 7fda12000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c10f2a0 con 0x7fda0c07ee40 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.056+0000 7fda0b400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.056+0000 7fda0b400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c19ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45002/0 (socket says 172.21.15.94:45002) 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.056+0000 7fda0b400700 1 -- 172.21.15.94:0/4089052331 learned_addr learned my addr 172.21.15.94:0/4089052331 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.056+0000 7fda0b400700 1 -- 172.21.15.94:0/4089052331 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0c19f360 con 0x7fda0c07ee40 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.056+0000 7fda0b400700 1 --2- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c19ee20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc004a60 tx=0x7fd9fc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.057+0000 7fda03e00700 1 -- 172.21.15.94:0/4089052331 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9fc020020 con 0x7fda0c07ee40 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.057+0000 7fda03e00700 1 -- 172.21.15.94:0/4089052331 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9fc012430 con 0x7fda0c07ee40 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.057+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda0c19f5f0 con 0x7fda0c07ee40 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.057+0000 7fda03e00700 1 -- 172.21.15.94:0/4089052331 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9fc017040 con 0x7fda0c07ee40 2024-04-26T19:59:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.057+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda0c1a22e0 con 0x7fda0c07ee40 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.058+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda0c064a80 con 0x7fda0c07ee40 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.058+0000 7fda03e00700 1 -- 172.21.15.94:0/4089052331 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd9fc023020 con 0x7fda0c07ee40 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.058+0000 7fda03e00700 1 --2- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fd9f0039600 0x7fd9f003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.058+0000 7fda03e00700 1 -- 172.21.15.94:0/4089052331 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd9fc00f050 con 0x7fda0c07ee40 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.059+0000 7fda0aa00700 1 --2- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fd9f0039600 0x7fd9f003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.059+0000 7fda0aa00700 1 --2- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fd9f0039600 0x7fd9f003bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8006fd0 tx=0x7fd9f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.066+0000 7fda03e00700 1 -- 172.21.15.94:0/4089052331 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd9fc02a020 con 0x7fda0c07ee40 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.309+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fda0c1054a0 con 0x7fd9f0039600 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.320+0000 7fda03e00700 1 -- 172.21.15.94:0/4089052331 <== mgr.14120 v2:172.21.15.94:6800/3651438916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fda0c1054a0 con 0x7fd9f0039600 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.323+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fd9f0039600 msgr2=0x7fd9f003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.323+0000 7fda12000700 1 --2- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fd9f0039600 0x7fd9f003bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8006fd0 tx=0x7fd9f8006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.323+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 msgr2=0x7fda0c19ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.323+0000 7fda12000700 1 --2- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c19ee20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc004a60 tx=0x7fd9fc00bd50 comp rx=0 tx=0).stop 2024-04-26T19:59:41.999 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.323+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 shutdown_connections 2024-04-26T19:59:42.000 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.323+0000 7fda12000700 1 --2- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fd9f0039600 0x7fd9f003bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:42.000 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.323+0000 7fda12000700 1 --2- 172.21.15.94:0/4089052331 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda0c07ee40 0x7fda0c19ee20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:42.000 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.323+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 >> 172.21.15.94:0/4089052331 conn(0x7fda0c1042f0 msgr2=0x7fda0c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:42.000 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.323+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 shutdown_connections 2024-04-26T19:59:42.000 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:41.324+0000 7fda12000700 1 -- 172.21.15.94:0/4089052331 wait complete. 2024-04-26T19:59:42.935 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:42 smithi094 ceph-mon[28050]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T19:59:43.640 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T19:59:43.640 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.691+0000 7f957fe00700 1 Processor -- start 2024-04-26T19:59:43.640 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.692+0000 7f957fe00700 1 -- start start 2024-04-26T19:59:43.640 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.692+0000 7f957fe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957807d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:43.640 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.692+0000 7f957fe00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f957807d870 con 0x7f957807ee40 2024-04-26T19:59:43.640 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.693+0000 7f957d400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957807d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:43.640 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.693+0000 7f957d400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957807d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45018/0 (socket says 172.21.15.94:45018) 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.693+0000 7f957d400700 1 -- 172.21.15.94:0/2699010494 learned_addr learned my addr 172.21.15.94:0/2699010494 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.693+0000 7f957d400700 1 -- 172.21.15.94:0/2699010494 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957807d9f0 con 0x7f957807ee40 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.694+0000 7f957d400700 1 --2- 172.21.15.94:0/2699010494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957807d2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f956c0098a0 tx=0x7f956c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4267383003cd6f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.694+0000 7f9577e00700 1 -- 172.21.15.94:0/2699010494 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f956c004160 con 0x7f957807ee40 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.694+0000 7f9577e00700 1 -- 172.21.15.94:0/2699010494 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f956c004300 con 0x7f957807ee40 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.694+0000 7f9577e00700 1 -- 172.21.15.94:0/2699010494 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f956c011af0 con 0x7f957807ee40 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.695+0000 7f957fe00700 1 -- 172.21.15.94:0/2699010494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 msgr2=0x7f957807d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.695+0000 7f957fe00700 1 --2- 172.21.15.94:0/2699010494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957807d2a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f956c0098a0 tx=0x7f956c009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.695+0000 7f957fe00700 1 -- 172.21.15.94:0/2699010494 shutdown_connections 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.695+0000 7f957fe00700 1 --2- 172.21.15.94:0/2699010494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957807d2a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.695+0000 7f957fe00700 1 -- 172.21.15.94:0/2699010494 >> 172.21.15.94:0/2699010494 conn(0x7f95781042f0 msgr2=0x7f9578106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.695+0000 7f957fe00700 1 -- 172.21.15.94:0/2699010494 shutdown_connections 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.695+0000 7f957fe00700 1 -- 172.21.15.94:0/2699010494 wait complete. 2024-04-26T19:59:43.641 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.696+0000 7f957fe00700 1 Processor -- start 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.696+0000 7f957fe00700 1 -- start start 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.697+0000 7f957fe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.697+0000 7f957fe00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f957810f2a0 con 0x7f957807ee40 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.697+0000 7f957d400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.697+0000 7f957d400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957819ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45022/0 (socket says 172.21.15.94:45022) 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.697+0000 7f957d400700 1 -- 172.21.15.94:0/843990854 learned_addr learned my addr 172.21.15.94:0/843990854 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.697+0000 7f957d400700 1 -- 172.21.15.94:0/843990854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957819f3a0 con 0x7f957807ee40 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.698+0000 7f957d400700 1 --2- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957819ee60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f956c0117b0 tx=0x7f956c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.698+0000 7f9576000700 1 -- 172.21.15.94:0/843990854 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f956c003af0 con 0x7f957807ee40 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.698+0000 7f9576000700 1 -- 172.21.15.94:0/843990854 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f956c020070 con 0x7f957807ee40 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.698+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957819f630 con 0x7f957807ee40 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.698+0000 7f9576000700 1 -- 172.21.15.94:0/843990854 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f956c019630 con 0x7f957807ee40 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.698+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95781a2320 con 0x7f957807ee40 2024-04-26T19:59:43.642 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.699+0000 7f9576000700 1 -- 172.21.15.94:0/843990854 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f956c01a9c0 con 0x7f957807ee40 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.699+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9578064a80 con 0x7f957807ee40 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.700+0000 7f9576000700 1 --2- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9560039600 0x7f956003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.700+0000 7f9576000700 1 -- 172.21.15.94:0/843990854 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f956c04db70 con 0x7f957807ee40 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.700+0000 7f957ca00700 1 --2- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9560039600 0x7f956003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.701+0000 7f957ca00700 1 --2- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9560039600 0x7f956003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9564006fd0 tx=0x7f9564006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.706+0000 7f9576000700 1 -- 172.21.15.94:0/843990854 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f956c00e260 con 0x7f957807ee40 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.935+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f95781054d0 con 0x7f9560039600 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.937+0000 7f9576000700 1 -- 172.21.15.94:0/843990854 <== mgr.14120 v2:172.21.15.94:6800/3651438916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f95781054d0 con 0x7f9560039600 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.939+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9560039600 msgr2=0x7f956003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.939+0000 7f957fe00700 1 --2- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9560039600 0x7f956003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9564006fd0 tx=0x7f9564006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.939+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 msgr2=0x7f957819ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.939+0000 7f957fe00700 1 --2- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957819ee60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f956c0117b0 tx=0x7f956c011b70 comp rx=0 tx=0).stop 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.940+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 shutdown_connections 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.940+0000 7f957fe00700 1 --2- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f9560039600 0x7f956003bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:43.643 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.940+0000 7f957fe00700 1 --2- 172.21.15.94:0/843990854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f957807ee40 0x7f957819ee60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:43.644 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.940+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 >> 172.21.15.94:0/843990854 conn(0x7f95781042f0 msgr2=0x7f9578104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:43.644 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.940+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 shutdown_connections 2024-04-26T19:59:43.644 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:42.940+0000 7f957fe00700 1 -- 172.21.15.94:0/843990854 wait complete. 2024-04-26T19:59:43.644 INFO:teuthology.orchestra.run.smithi094.stdout:Generating ssh key... 2024-04-26T19:59:43.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:43 smithi094 ceph-mon[28050]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: Generating public/private rsa key pair. 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: Your identification has been saved in /tmp/tmpuc68qxmk/key. 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: Your public key has been saved in /tmp/tmpuc68qxmk/key.pub. 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: The key fingerprint is: 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: SHA256:YLC3Cwp6hRMt6K1XwMUJVQVfLf44aWncjYEyqEbrWnk ceph-420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: The key's randomart image is: 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: +---[RSA 3072]----+ 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: | .=oooo. .. | 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: | ....= . .. . | 2024-04-26T19:59:44.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: |. ooo + ... o | 2024-04-26T19:59:44.989 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: |. .+.+ + o o . | 2024-04-26T19:59:44.989 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: |..o.+.+ S + * + | 2024-04-26T19:59:44.989 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: |...+.=.. O + . | 2024-04-26T19:59:44.989 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: |..o.oo.E o . | 2024-04-26T19:59:44.989 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: | .. ... | 2024-04-26T19:59:44.989 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: | .. | 2024-04-26T19:59:44.989 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 19:59:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: +----[SHA256]-----+ 2024-04-26T19:59:45.364 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.227+0000 7fde85e00700 1 Processor -- start 2024-04-26T19:59:45.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.227+0000 7fde85e00700 1 -- start start 2024-04-26T19:59:45.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.227+0000 7fde85e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80109590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:45.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.227+0000 7fde85e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fde801076b0 con 0x7fde80109170 2024-04-26T19:59:45.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.228+0000 7fde84a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80109590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:45.369 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.228+0000 7fde84a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80109590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45038/0 (socket says 172.21.15.94:45038) 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.228+0000 7fde84a00700 1 -- 172.21.15.94:0/3885552263 learned_addr learned my addr 172.21.15.94:0/3885552263 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.229+0000 7fde84a00700 1 -- 172.21.15.94:0/3885552263 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde80107830 con 0x7fde80109170 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.229+0000 7fde84a00700 1 --2- 172.21.15.94:0/3885552263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80109590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde70009e80 tx=0x7fde7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bde9f9efa0aec63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.229+0000 7fde7fe00700 1 -- 172.21.15.94:0/3885552263 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde70017070 con 0x7fde80109170 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.229+0000 7fde7fe00700 1 -- 172.21.15.94:0/3885552263 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fde7000b8c0 con 0x7fde80109170 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.229+0000 7fde7fe00700 1 -- 172.21.15.94:0/3885552263 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde70004490 con 0x7fde80109170 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.230+0000 7fde85e00700 1 -- 172.21.15.94:0/3885552263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 msgr2=0x7fde80109590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.230+0000 7fde85e00700 1 --2- 172.21.15.94:0/3885552263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80109590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde70009e80 tx=0x7fde7000b330 comp rx=0 tx=0).stop 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.230+0000 7fde85e00700 1 -- 172.21.15.94:0/3885552263 shutdown_connections 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.230+0000 7fde85e00700 1 --2- 172.21.15.94:0/3885552263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80109590 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.230+0000 7fde85e00700 1 -- 172.21.15.94:0/3885552263 >> 172.21.15.94:0/3885552263 conn(0x7fde80103070 msgr2=0x7fde80105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.230+0000 7fde85e00700 1 -- 172.21.15.94:0/3885552263 shutdown_connections 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.230+0000 7fde85e00700 1 -- 172.21.15.94:0/3885552263 wait complete. 2024-04-26T19:59:45.370 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.231+0000 7fde85e00700 1 Processor -- start 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.232+0000 7fde85e00700 1 -- start start 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.232+0000 7fde85e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80192cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.232+0000 7fde85e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fde8010c040 con 0x7fde80109170 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.232+0000 7fde84a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80192cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.232+0000 7fde84a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80192cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45050/0 (socket says 172.21.15.94:45050) 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.232+0000 7fde84a00700 1 -- 172.21.15.94:0/536316264 learned_addr learned my addr 172.21.15.94:0/536316264 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.233+0000 7fde84a00700 1 -- 172.21.15.94:0/536316264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde80193200 con 0x7fde80109170 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.233+0000 7fde84a00700 1 --2- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80192cc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde70004a60 tx=0x7fde7000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.233+0000 7fde7e000700 1 -- 172.21.15.94:0/536316264 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde70020020 con 0x7fde80109170 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.233+0000 7fde7e000700 1 -- 172.21.15.94:0/536316264 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fde70012430 con 0x7fde80109170 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.233+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde801913f0 con 0x7fde80109170 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.233+0000 7fde7e000700 1 -- 172.21.15.94:0/536316264 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde70017040 con 0x7fde80109170 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.233+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde80191930 con 0x7fde80109170 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.235+0000 7fde7e000700 1 -- 172.21.15.94:0/536316264 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fde70023020 con 0x7fde80109170 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.235+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8010c1c0 con 0x7fde80109170 2024-04-26T19:59:45.371 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.235+0000 7fde7e000700 1 --2- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fde64039600 0x7fde6403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.235+0000 7fde7e000700 1 -- 172.21.15.94:0/536316264 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fde7000f050 con 0x7fde80109170 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.235+0000 7fde77e00700 1 --2- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fde64039600 0x7fde6403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.236+0000 7fde77e00700 1 --2- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fde64039600 0x7fde6403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde6c006fd0 tx=0x7fde6c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.241+0000 7fde7e000700 1 -- 172.21.15.94:0/536316264 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fde7002a020 con 0x7fde80109170 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.468+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fde801969c0 con 0x7fde64039600 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.697+0000 7fde7e000700 1 -- 172.21.15.94:0/536316264 <== mgr.14120 v2:172.21.15.94:6800/3651438916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fde801969c0 con 0x7fde64039600 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.700+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fde64039600 msgr2=0x7fde6403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.700+0000 7fde85e00700 1 --2- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fde64039600 0x7fde6403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde6c006fd0 tx=0x7fde6c006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.700+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 msgr2=0x7fde80192cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.700+0000 7fde85e00700 1 --2- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80192cc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde70004a60 tx=0x7fde7000bd50 comp rx=0 tx=0).stop 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.701+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 shutdown_connections 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.701+0000 7fde85e00700 1 --2- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fde64039600 0x7fde6403bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.701+0000 7fde85e00700 1 --2- 172.21.15.94:0/536316264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde80109170 0x7fde80192cc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.701+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 >> 172.21.15.94:0/536316264 conn(0x7fde80103070 msgr2=0x7fde80103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.701+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 shutdown_connections 2024-04-26T19:59:45.372 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:44.701+0000 7fde85e00700 1 -- 172.21.15.94:0/536316264 wait complete. 2024-04-26T19:59:45.979 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:45 smithi094 ceph-mon[28050]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T19:59:45.979 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:45 smithi094 ceph-mon[28050]: Generating ssh key... 2024-04-26T19:59:45.979 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:45 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:45.979 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:45 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:46.916 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdGNxZAv1Ik/+3ASg9HX0/PijaT4004dLmquyvCawUgYDK+O92hWCzvd4zY88TFUzKCBypQnqubA+FumYVeBXr76ZikWVgz67jS+EEY56QxVCEqjtXah1NQEEpRgtKkt0I9ZAW8d/ozNQqhXXO7hURnH2jaB4U1QZ1QpiHtDCbUHXU5267b/K5CWVE3b1EOPas/2IxjwVKcAKr1d51208EOcpAsmUpx57DMQiLUuXEkmwybpOMOnflErzglMx/kwKcbAvzDW88IkVE29A4f2PT048sCauM5I/pu/j1hG3CJZPH6D+j5IW+bWx2muHEA2pizMzIepQJtdCBQMO7FH3saHGEZkpVFEPHTMivNjL16/xFHOBz4e/fz4A+lXidWsaCahyFAzJo0iDdbKO4H00UvY8b/dhSz0JOGjjxUYllHabAKvV6H3XrCk2tzoT++hevT21s2o/MktM/RR7ArF3y4gbCDqqBvnOscQmXKMkGZszYcI3vHAxTLoiW7t2N6OM= ceph-420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T19:59:46.917 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.978+0000 7f2820c00700 1 Processor -- start 2024-04-26T19:59:46.917 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.978+0000 7f2820c00700 1 -- start start 2024-04-26T19:59:46.917 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.978+0000 7f2820c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c10bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:46.917 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.979+0000 7f2820c00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f281c07ce30 con 0x7f281c109790 2024-04-26T19:59:46.917 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.979+0000 7f2819e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c10bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:46.917 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.979+0000 7f2819e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c10bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45066/0 (socket says 172.21.15.94:45066) 2024-04-26T19:59:46.917 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.979+0000 7f2819e00700 1 -- 172.21.15.94:0/3349415780 learned_addr learned my addr 172.21.15.94:0/3349415780 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:46.917 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.979+0000 7f2819e00700 1 -- 172.21.15.94:0/3349415780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281c10c0c0 con 0x7f281c109790 2024-04-26T19:59:46.921 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.980+0000 7f2819e00700 1 --2- 172.21.15.94:0/3349415780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c10bb80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f280c0098a0 tx=0x7f280c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8fc2a69629c001e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:46.921 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.980+0000 7f2818a00700 1 -- 172.21.15.94:0/3349415780 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f280c004160 con 0x7f281c109790 2024-04-26T19:59:46.921 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.980+0000 7f2818a00700 1 -- 172.21.15.94:0/3349415780 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f280c004300 con 0x7f281c109790 2024-04-26T19:59:46.921 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.980+0000 7f2818a00700 1 -- 172.21.15.94:0/3349415780 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f280c011af0 con 0x7f281c109790 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.981+0000 7f2820c00700 1 -- 172.21.15.94:0/3349415780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 msgr2=0x7f281c10bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.981+0000 7f2820c00700 1 --2- 172.21.15.94:0/3349415780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c10bb80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f280c0098a0 tx=0x7f280c009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.981+0000 7f2820c00700 1 -- 172.21.15.94:0/3349415780 shutdown_connections 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.981+0000 7f2820c00700 1 --2- 172.21.15.94:0/3349415780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c10bb80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.981+0000 7f2820c00700 1 -- 172.21.15.94:0/3349415780 >> 172.21.15.94:0/3349415780 conn(0x7f281c1031e0 msgr2=0x7f281c105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.981+0000 7f2820c00700 1 -- 172.21.15.94:0/3349415780 shutdown_connections 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.981+0000 7f2820c00700 1 -- 172.21.15.94:0/3349415780 wait complete. 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.982+0000 7f2820c00700 1 Processor -- start 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.982+0000 7f2820c00700 1 -- start start 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.983+0000 7f2820c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.983+0000 7f2820c00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f281c10d090 con 0x7f281c109790 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.983+0000 7f2819e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.983+0000 7f2819e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c19eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45074/0 (socket says 172.21.15.94:45074) 2024-04-26T19:59:46.922 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.983+0000 7f2819e00700 1 -- 172.21.15.94:0/1169873214 learned_addr learned my addr 172.21.15.94:0/1169873214 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.983+0000 7f2819e00700 1 -- 172.21.15.94:0/1169873214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281c19f3e0 con 0x7f281c109790 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.984+0000 7f2819e00700 1 --2- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c19eea0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f280c011ac0 tx=0x7f280c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.984+0000 7f2812a00700 1 -- 172.21.15.94:0/1169873214 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f280c003970 con 0x7f281c109790 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.984+0000 7f2812a00700 1 -- 172.21.15.94:0/1169873214 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f280c003b10 con 0x7f281c109790 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.984+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f281c19f670 con 0x7f281c109790 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.984+0000 7f2812a00700 1 -- 172.21.15.94:0/1169873214 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f280c024610 con 0x7f281c109790 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.984+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f281c1a2360 con 0x7f281c109790 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.985+0000 7f2812a00700 1 -- 172.21.15.94:0/1169873214 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f280c01a9e0 con 0x7f281c109790 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.985+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f281c064a80 con 0x7f281c109790 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.986+0000 7f2812a00700 1 --2- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f28000395b0 0x7f280003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.986+0000 7f2812a00700 1 -- 172.21.15.94:0/1169873214 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f280c04d940 con 0x7f281c109790 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.986+0000 7f2819400700 1 --2- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f28000395b0 0x7f280003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.987+0000 7f2819400700 1 --2- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f28000395b0 0x7f280003ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2808006fd0 tx=0x7f2808006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:46.923 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:45.991+0000 7f2812a00700 1 -- 172.21.15.94:0/1169873214 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f280c00b3e0 con 0x7f281c109790 2024-04-26T19:59:46.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.216+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f281c1043a0 con 0x7f28000395b0 2024-04-26T19:59:46.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.217+0000 7f2812a00700 1 -- 172.21.15.94:0/1169873214 <== mgr.14120 v2:172.21.15.94:6800/3651438916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f281c1043a0 con 0x7f28000395b0 2024-04-26T19:59:46.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.220+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f28000395b0 msgr2=0x7f280003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:46.924 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.220+0000 7f2820c00700 1 --2- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f28000395b0 0x7f280003ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2808006fd0 tx=0x7f2808006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:46.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.220+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 msgr2=0x7f281c19eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:46.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.220+0000 7f2820c00700 1 --2- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c19eea0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f280c011ac0 tx=0x7f280c011ba0 comp rx=0 tx=0).stop 2024-04-26T19:59:46.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.220+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 shutdown_connections 2024-04-26T19:59:46.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.220+0000 7f2820c00700 1 --2- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f28000395b0 0x7f280003ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:46.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.220+0000 7f2820c00700 1 --2- 172.21.15.94:0/1169873214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c109790 0x7f281c19eea0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:46.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.220+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 >> 172.21.15.94:0/1169873214 conn(0x7f281c1031e0 msgr2=0x7f281c103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:46.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.220+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 shutdown_connections 2024-04-26T19:59:46.925 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:46.221+0000 7f2820c00700 1 -- 172.21.15.94:0/1169873214 wait complete. 2024-04-26T19:59:46.926 INFO:teuthology.orchestra.run.smithi094.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T19:59:46.926 INFO:teuthology.orchestra.run.smithi094.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T19:59:46.934 INFO:teuthology.orchestra.run.smithi094.stdout:Adding host smithi094... 2024-04-26T19:59:47.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:47 smithi094 ceph-mon[28050]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T19:59:48.180 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:48 smithi094 ceph-mon[28050]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi094", "addr": "172.21.15.94", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T19:59:49.329 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:49 smithi094 ceph-mon[28050]: Deploying cephadm binary to smithi094 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Added host 'smithi094' with addr '172.21.15.94' 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.552+0000 7f2aed800700 1 Processor -- start 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.552+0000 7f2aed800700 1 -- start start 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.553+0000 7f2aed800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae8109100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.553+0000 7f2aed800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae81096d0 con 0x7f2ae8108ce0 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.553+0000 7f2ae6a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae8109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.553+0000 7f2ae6a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae8109100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45082/0 (socket says 172.21.15.94:45082) 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.553+0000 7f2ae6a00700 1 -- 172.21.15.94:0/1042052462 learned_addr learned my addr 172.21.15.94:0/1042052462 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.554+0000 7f2ae6a00700 1 -- 172.21.15.94:0/1042052462 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae8109ee0 con 0x7f2ae8108ce0 2024-04-26T19:59:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.554+0000 7f2ae6a00700 1 --2- 172.21.15.94:0/1042052462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae8109100 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80098a0 tx=0x7f2ad8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9aa881ecef5af8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.555+0000 7f2ae5600700 1 -- 172.21.15.94:0/1042052462 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad8004160 con 0x7f2ae8108ce0 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.555+0000 7f2ae5600700 1 -- 172.21.15.94:0/1042052462 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2ad8004300 con 0x7f2ae8108ce0 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.555+0000 7f2ae5600700 1 -- 172.21.15.94:0/1042052462 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad8011af0 con 0x7f2ae8108ce0 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.555+0000 7f2aed800700 1 -- 172.21.15.94:0/1042052462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 msgr2=0x7f2ae8109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.555+0000 7f2aed800700 1 --2- 172.21.15.94:0/1042052462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae8109100 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80098a0 tx=0x7f2ad8009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.555+0000 7f2aed800700 1 -- 172.21.15.94:0/1042052462 shutdown_connections 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.555+0000 7f2aed800700 1 --2- 172.21.15.94:0/1042052462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae8109100 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.555+0000 7f2aed800700 1 -- 172.21.15.94:0/1042052462 >> 172.21.15.94:0/1042052462 conn(0x7f2ae81042a0 msgr2=0x7f2ae81066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.556+0000 7f2aed800700 1 -- 172.21.15.94:0/1042052462 shutdown_connections 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.556+0000 7f2aed800700 1 -- 172.21.15.94:0/1042052462 wait complete. 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.557+0000 7f2aed800700 1 Processor -- start 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.557+0000 7f2aed800700 1 -- start start 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.557+0000 7f2aed800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.557+0000 7f2aed800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae810d290 con 0x7f2ae8108ce0 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.557+0000 7f2ae6a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.557+0000 7f2ae6a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45088/0 (socket says 172.21.15.94:45088) 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.557+0000 7f2ae6a00700 1 -- 172.21.15.94:0/1059116866 learned_addr learned my addr 172.21.15.94:0/1059116866 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.558+0000 7f2ae6a00700 1 -- 172.21.15.94:0/1059116866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae819f400 con 0x7f2ae8108ce0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.558+0000 7f2ae6a00700 1 --2- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae819eec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80117b0 tx=0x7f2ad8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.558+0000 7f2adf400700 1 -- 172.21.15.94:0/1059116866 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad8003af0 con 0x7f2ae8108ce0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.558+0000 7f2adf400700 1 -- 172.21.15.94:0/1059116866 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2ad8020070 con 0x7f2ae8108ce0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.558+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae819f690 con 0x7f2ae8108ce0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.558+0000 7f2adf400700 1 -- 172.21.15.94:0/1059116866 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad8019630 con 0x7f2ae8108ce0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.558+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae81a2380 con 0x7f2ae8108ce0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.559+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae8064a80 con 0x7f2ae8108ce0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.560+0000 7f2adf400700 1 -- 172.21.15.94:0/1059116866 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2ad801a9c0 con 0x7f2ae8108ce0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.560+0000 7f2adf400700 1 --2- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f2acc039600 0x7f2acc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.560+0000 7f2adf400700 1 -- 172.21.15.94:0/1059116866 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2ad804dbf0 con 0x7f2ae8108ce0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.561+0000 7f2ae6000700 1 --2- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f2acc039600 0x7f2acc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:54.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.561+0000 7f2ae6000700 1 --2- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f2acc039600 0x7f2acc03bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4006fd0 tx=0x7f2ad4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.566+0000 7f2adf400700 1 -- 172.21.15.94:0/1059116866 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2ad80042a0 con 0x7f2ae8108ce0 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:47.794+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi094", "addr": "172.21.15.94", "target": ["mon-mgr", ""]}) -- 0x7f2ae8105460 con 0x7f2acc039600 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.888+0000 7f2adf400700 1 -- 172.21.15.94:0/1059116866 <== mgr.14120 v2:172.21.15.94:6800/3651438916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f2ae8105460 con 0x7f2acc039600 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.892+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f2acc039600 msgr2=0x7f2acc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.892+0000 7f2aed800700 1 --2- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f2acc039600 0x7f2acc03bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4006fd0 tx=0x7f2ad4006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.892+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 msgr2=0x7f2ae819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.892+0000 7f2aed800700 1 --2- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae819eec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80117b0 tx=0x7f2ad8011b70 comp rx=0 tx=0).stop 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.892+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 shutdown_connections 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.892+0000 7f2aed800700 1 --2- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f2acc039600 0x7f2acc03bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.892+0000 7f2aed800700 1 --2- 172.21.15.94:0/1059116866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae8108ce0 0x7f2ae819eec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.893+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 >> 172.21.15.94:0/1059116866 conn(0x7f2ae81042a0 msgr2=0x7f2ae8104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.893+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 shutdown_connections 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:53.893+0000 7f2aed800700 1 -- 172.21.15.94:0/1059116866 wait complete. 2024-04-26T19:59:54.603 INFO:teuthology.orchestra.run.smithi094.stdout:Deploying unmanaged mon service... 2024-04-26T19:59:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:54 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:54 smithi094 ceph-mon[28050]: Added host smithi094 2024-04-26T19:59:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:54 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T19:59:56.332 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T19:59:56.333 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.283+0000 7f582b800700 1 Processor -- start 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.283+0000 7f582b800700 1 -- start start 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.284+0000 7f582b800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582407d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.284+0000 7f582b800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f582407d850 con 0x7f582407ee20 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.284+0000 7f5828e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582407d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.284+0000 7f5828e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582407d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60792/0 (socket says 172.21.15.94:60792) 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.284+0000 7f5828e00700 1 -- 172.21.15.94:0/133016593 learned_addr learned my addr 172.21.15.94:0/133016593 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.285+0000 7f5828e00700 1 -- 172.21.15.94:0/133016593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f582407d9d0 con 0x7f582407ee20 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.285+0000 7f5828e00700 1 --2- 172.21.15.94:0/133016593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582407d280 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58180098a0 tx=0x7f5818009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb878f5d33b9778 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.285+0000 7f5823400700 1 -- 172.21.15.94:0/133016593 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5818004160 con 0x7f582407ee20 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.285+0000 7f5823400700 1 -- 172.21.15.94:0/133016593 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5818004300 con 0x7f582407ee20 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.285+0000 7f5823400700 1 -- 172.21.15.94:0/133016593 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5818011af0 con 0x7f582407ee20 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.286+0000 7f582b800700 1 -- 172.21.15.94:0/133016593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 msgr2=0x7f582407d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:56.334 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.286+0000 7f582b800700 1 --2- 172.21.15.94:0/133016593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582407d280 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58180098a0 tx=0x7f5818009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:56.337 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.286+0000 7f582b800700 1 -- 172.21.15.94:0/133016593 shutdown_connections 2024-04-26T19:59:56.337 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.286+0000 7f582b800700 1 --2- 172.21.15.94:0/133016593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582407d280 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.286+0000 7f582b800700 1 -- 172.21.15.94:0/133016593 >> 172.21.15.94:0/133016593 conn(0x7f58241042a0 msgr2=0x7f58241066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.286+0000 7f582b800700 1 -- 172.21.15.94:0/133016593 shutdown_connections 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.286+0000 7f582b800700 1 -- 172.21.15.94:0/133016593 wait complete. 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.287+0000 7f582b800700 1 Processor -- start 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.288+0000 7f582b800700 1 -- start start 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.288+0000 7f582b800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.288+0000 7f582b800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f582410f290 con 0x7f582407ee20 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.288+0000 7f5828e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.288+0000 7f5828e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582419eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60794/0 (socket says 172.21.15.94:60794) 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.288+0000 7f5828e00700 1 -- 172.21.15.94:0/166255329 learned_addr learned my addr 172.21.15.94:0/166255329 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.289+0000 7f5828e00700 1 -- 172.21.15.94:0/166255329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f582419f3e0 con 0x7f582407ee20 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.289+0000 7f5828e00700 1 --2- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582419eea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5818009fb0 tx=0x7f5818011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.289+0000 7f5821600700 1 -- 172.21.15.94:0/166255329 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58180036a0 con 0x7f582407ee20 2024-04-26T19:59:56.338 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.289+0000 7f5821600700 1 -- 172.21.15.94:0/166255329 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f581801a430 con 0x7f582407ee20 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.289+0000 7f5821600700 1 -- 172.21.15.94:0/166255329 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5818019550 con 0x7f582407ee20 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.289+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f582419f670 con 0x7f582407ee20 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.289+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58241a2360 con 0x7f582407ee20 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.290+0000 7f5821600700 1 -- 172.21.15.94:0/166255329 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5818003840 con 0x7f582407ee20 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.290+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5824064a80 con 0x7f582407ee20 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.291+0000 7f5821600700 1 --2- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f580c0395b0 0x7f580c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.291+0000 7f5821600700 1 -- 172.21.15.94:0/166255329 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f581804d920 con 0x7f582407ee20 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.291+0000 7f5823e00700 1 --2- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f580c0395b0 0x7f580c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.292+0000 7f5823e00700 1 --2- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f580c0395b0 0x7f580c03ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5810006fd0 tx=0x7f5810006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.297+0000 7f5821600700 1 -- 172.21.15.94:0/166255329 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f581804d270 con 0x7f582407ee20 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.552+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f58241055b0 con 0x7f580c0395b0 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.561+0000 7f5821600700 1 -- 172.21.15.94:0/166255329 <== mgr.14120 v2:172.21.15.94:6800/3651438916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f58241055b0 con 0x7f580c0395b0 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.564+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f580c0395b0 msgr2=0x7f580c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:56.339 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.564+0000 7f582b800700 1 --2- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f580c0395b0 0x7f580c03ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5810006fd0 tx=0x7f5810006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:56.340 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.564+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 msgr2=0x7f582419eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:56.341 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.564+0000 7f582b800700 1 --2- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582419eea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5818009fb0 tx=0x7f5818011940 comp rx=0 tx=0).stop 2024-04-26T19:59:56.341 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.564+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 shutdown_connections 2024-04-26T19:59:56.341 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.564+0000 7f582b800700 1 --2- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f580c0395b0 0x7f580c03ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:56.341 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.564+0000 7f582b800700 1 --2- 172.21.15.94:0/166255329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f582407ee20 0x7f582419eea0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:56.341 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.564+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 >> 172.21.15.94:0/166255329 conn(0x7f58241042a0 msgr2=0x7f5824104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:56.341 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.565+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 shutdown_connections 2024-04-26T19:59:56.341 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:55.565+0000 7f582b800700 1 -- 172.21.15.94:0/166255329 wait complete. 2024-04-26T19:59:56.341 INFO:teuthology.orchestra.run.smithi094.stdout:Deploying unmanaged mgr service... 2024-04-26T19:59:56.605 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:56 smithi094 ceph-mon[28050]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T19:59:56.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:56 smithi094 ceph-mon[28050]: Saving service mon spec with placement count:5 2024-04-26T19:59:56.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:56 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.006+0000 7fe58ca00700 1 Processor -- start 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.007+0000 7fe58ca00700 1 -- start start 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.007+0000 7fe58ca00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe588109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.007+0000 7fe58ca00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe588109700 con 0x7fe588108d10 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.008+0000 7fe585e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe588109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.008+0000 7fe585e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe588109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60810/0 (socket says 172.21.15.94:60810) 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.008+0000 7fe585e00700 1 -- 172.21.15.94:0/3878076881 learned_addr learned my addr 172.21.15.94:0/3878076881 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.008+0000 7fe585e00700 1 -- 172.21.15.94:0/3878076881 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe588109f10 con 0x7fe588108d10 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.009+0000 7fe585e00700 1 --2- 172.21.15.94:0/3878076881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe588109130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5780098a0 tx=0x7fe578009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cebbfc66f4830f52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.009+0000 7fe584a00700 1 -- 172.21.15.94:0/3878076881 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe578004160 con 0x7fe588108d10 2024-04-26T19:59:57.945 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.009+0000 7fe584a00700 1 -- 172.21.15.94:0/3878076881 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe578004300 con 0x7fe588108d10 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.009+0000 7fe584a00700 1 -- 172.21.15.94:0/3878076881 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe578011af0 con 0x7fe588108d10 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.010+0000 7fe58ca00700 1 -- 172.21.15.94:0/3878076881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 msgr2=0x7fe588109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.010+0000 7fe58ca00700 1 --2- 172.21.15.94:0/3878076881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe588109130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5780098a0 tx=0x7fe578009bb0 comp rx=0 tx=0).stop 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.010+0000 7fe58ca00700 1 -- 172.21.15.94:0/3878076881 shutdown_connections 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.010+0000 7fe58ca00700 1 --2- 172.21.15.94:0/3878076881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe588109130 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.010+0000 7fe58ca00700 1 -- 172.21.15.94:0/3878076881 >> 172.21.15.94:0/3878076881 conn(0x7fe5881042f0 msgr2=0x7fe588106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.010+0000 7fe58ca00700 1 -- 172.21.15.94:0/3878076881 shutdown_connections 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.010+0000 7fe58ca00700 1 -- 172.21.15.94:0/3878076881 wait complete. 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.012+0000 7fe58ca00700 1 Processor -- start 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.012+0000 7fe58ca00700 1 -- start start 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.012+0000 7fe58ca00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe58807eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.012+0000 7fe58ca00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe58810d2c0 con 0x7fe588108d10 2024-04-26T19:59:57.946 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.013+0000 7fe585e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe58807eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.013+0000 7fe585e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe58807eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60814/0 (socket says 172.21.15.94:60814) 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.013+0000 7fe585e00700 1 -- 172.21.15.94:0/1929081489 learned_addr learned my addr 172.21.15.94:0/1929081489 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.013+0000 7fe585e00700 1 -- 172.21.15.94:0/1929081489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58807f030 con 0x7fe588108d10 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.013+0000 7fe585e00700 1 --2- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe58807eaf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe578009fb0 tx=0x7fe578011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.014+0000 7fe57ea00700 1 -- 172.21.15.94:0/1929081489 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5780036a0 con 0x7fe588108d10 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.014+0000 7fe57ea00700 1 -- 172.21.15.94:0/1929081489 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe57801a430 con 0x7fe588108d10 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.014+0000 7fe57ea00700 1 -- 172.21.15.94:0/1929081489 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe578019550 con 0x7fe588108d10 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.014+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58807d220 con 0x7fe588108d10 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.014+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe58807d760 con 0x7fe588108d10 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.015+0000 7fe57ea00700 1 -- 172.21.15.94:0/1929081489 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe57801a8e0 con 0x7fe588108d10 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.015+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe588064a80 con 0x7fe588108d10 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.015+0000 7fe57ea00700 1 --2- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fe56c0395b0 0x7fe56c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.015+0000 7fe57ea00700 1 -- 172.21.15.94:0/1929081489 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe57804d8d0 con 0x7fe588108d10 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.016+0000 7fe585400700 1 --2- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fe56c0395b0 0x7fe56c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.016+0000 7fe585400700 1 --2- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fe56c0395b0 0x7fe56c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe574006fd0 tx=0x7fe574006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.021+0000 7fe57ea00700 1 -- 172.21.15.94:0/1929081489 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe57801a690 con 0x7fe588108d10 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.267+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fe58807dfc0 con 0x7fe56c0395b0 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.275+0000 7fe57ea00700 1 -- 172.21.15.94:0/1929081489 <== mgr.14120 v2:172.21.15.94:6800/3651438916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe58807dfc0 con 0x7fe56c0395b0 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fe56c0395b0 msgr2=0x7fe56c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 --2- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fe56c0395b0 0x7fe56c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe574006fd0 tx=0x7fe574006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 msgr2=0x7fe58807eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 --2- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe58807eaf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe578009fb0 tx=0x7fe578011940 comp rx=0 tx=0).stop 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 shutdown_connections 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 --2- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fe56c0395b0 0x7fe56c03ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 --2- 172.21.15.94:0/1929081489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe588108d10 0x7fe58807eaf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 >> 172.21.15.94:0/1929081489 conn(0x7fe5881042f0 msgr2=0x7fe588104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 shutdown_connections 2024-04-26T19:59:57.948 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:57.278+0000 7fe58ca00700 1 -- 172.21.15.94:0/1929081489 wait complete. 2024-04-26T19:59:58.528 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:58 smithi094 ceph-mon[28050]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T19:59:58.528 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:58 smithi094 ceph-mon[28050]: Saving service mgr spec with placement count:2 2024-04-26T19:59:58.528 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:58 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T19:59:59.571 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.616+0000 7fc69be00700 1 Processor -- start 2024-04-26T19:59:59.571 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.617+0000 7fc69be00700 1 -- start start 2024-04-26T19:59:59.571 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.617+0000 7fc69be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c07d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.617+0000 7fc69be00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c07efd0 con 0x7fc69c07d2a0 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.617+0000 7fc69aa00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c07d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.617+0000 7fc69aa00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c07d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60826/0 (socket says 172.21.15.94:60826) 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.617+0000 7fc69aa00700 1 -- 172.21.15.94:0/3871496724 learned_addr learned my addr 172.21.15.94:0/3871496724 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.618+0000 7fc69aa00700 1 -- 172.21.15.94:0/3871496724 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c07dc00 con 0x7fc69c07d2a0 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.618+0000 7fc69aa00700 1 --2- 172.21.15.94:0/3871496724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c07d6c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009e80 tx=0x7fc68c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cb3c058078e365 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.618+0000 7fc69a000700 1 -- 172.21.15.94:0/3871496724 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc68c017070 con 0x7fc69c07d2a0 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.618+0000 7fc69a000700 1 -- 172.21.15.94:0/3871496724 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc68c00b8c0 con 0x7fc69c07d2a0 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.618+0000 7fc69a000700 1 -- 172.21.15.94:0/3871496724 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc68c004490 con 0x7fc69c07d2a0 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.619+0000 7fc69be00700 1 -- 172.21.15.94:0/3871496724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 msgr2=0x7fc69c07d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.619+0000 7fc69be00700 1 --2- 172.21.15.94:0/3871496724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c07d6c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009e80 tx=0x7fc68c00b330 comp rx=0 tx=0).stop 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.619+0000 7fc69be00700 1 -- 172.21.15.94:0/3871496724 shutdown_connections 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.619+0000 7fc69be00700 1 --2- 172.21.15.94:0/3871496724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c07d6c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.619+0000 7fc69be00700 1 -- 172.21.15.94:0/3871496724 >> 172.21.15.94:0/3871496724 conn(0x7fc69c104150 msgr2=0x7fc69c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.619+0000 7fc69be00700 1 -- 172.21.15.94:0/3871496724 shutdown_connections 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.619+0000 7fc69be00700 1 -- 172.21.15.94:0/3871496724 wait complete. 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.620+0000 7fc69be00700 1 Processor -- start 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.621+0000 7fc69be00700 1 -- start start 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.621+0000 7fc69be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.621+0000 7fc69be00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c10f220 con 0x7fc69c07d2a0 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.621+0000 7fc69aa00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.621+0000 7fc69aa00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60842/0 (socket says 172.21.15.94:60842) 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.621+0000 7fc69aa00700 1 -- 172.21.15.94:0/2584232615 learned_addr learned my addr 172.21.15.94:0/2584232615 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.622+0000 7fc69aa00700 1 -- 172.21.15.94:0/2584232615 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c19f2f0 con 0x7fc69c07d2a0 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.622+0000 7fc69aa00700 1 --2- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c19edb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00b7b0 tx=0x7fc68c00be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.622+0000 7fc693e00700 1 -- 172.21.15.94:0/2584232615 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc68c020020 con 0x7fc69c07d2a0 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.622+0000 7fc693e00700 1 -- 172.21.15.94:0/2584232615 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc68c003d10 con 0x7fc69c07d2a0 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.622+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c19f580 con 0x7fc69c07d2a0 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.622+0000 7fc693e00700 1 -- 172.21.15.94:0/2584232615 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc68c017040 con 0x7fc69c07d2a0 2024-04-26T19:59:59.573 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.623+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c1a2270 con 0x7fc69c07d2a0 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.624+0000 7fc693e00700 1 -- 172.21.15.94:0/2584232615 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc68c023020 con 0x7fc69c07d2a0 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.624+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc69c064a80 con 0x7fc69c07d2a0 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.624+0000 7fc693e00700 1 --2- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fc680039600 0x7fc68003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.624+0000 7fc693e00700 1 -- 172.21.15.94:0/2584232615 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc68c00f050 con 0x7fc69c07d2a0 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.624+0000 7fc692000700 1 --2- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fc680039600 0x7fc68003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.625+0000 7fc692000700 1 --2- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fc680039600 0x7fc68003bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc684006fd0 tx=0x7fc684006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.629+0000 7fc693e00700 1 -- 172.21.15.94:0/2584232615 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc68c02a020 con 0x7fc69c07d2a0 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.857+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fc69c1a27b0 con 0x7fc69c07d2a0 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.862+0000 7fc693e00700 1 -- 172.21.15.94:0/2584232615 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7fc68c02d350 con 0x7fc69c07d2a0 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.865+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fc680039600 msgr2=0x7fc68003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.865+0000 7fc69be00700 1 --2- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fc680039600 0x7fc68003bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc684006fd0 tx=0x7fc684006e40 comp rx=0 tx=0).stop 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.865+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 msgr2=0x7fc69c19edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T19:59:59.574 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.865+0000 7fc69be00700 1 --2- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c19edb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00b7b0 tx=0x7fc68c00be90 comp rx=0 tx=0).stop 2024-04-26T19:59:59.575 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.866+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 shutdown_connections 2024-04-26T19:59:59.575 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.866+0000 7fc69be00700 1 --2- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fc680039600 0x7fc68003bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:59.575 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.866+0000 7fc69be00700 1 --2- 172.21.15.94:0/2584232615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc69c07d2a0 0x7fc69c19edb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T19:59:59.575 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.866+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 >> 172.21.15.94:0/2584232615 conn(0x7fc69c104150 msgr2=0x7fc69c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T19:59:59.575 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.866+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 shutdown_connections 2024-04-26T19:59:59.575 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T19:59:58.866+0000 7fc69be00700 1 -- 172.21.15.94:0/2584232615 wait complete. 2024-04-26T20:00:00.128 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:59 smithi094 ceph-mon[28050]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:00.128 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:59 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2584232615' entity='client.admin' 2024-04-26T20:00:00.129 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:59 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T20:00:00.129 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 19:59:59 smithi094 ceph-mon[28050]: from='mgr.14120 172.21.15.94:0/1653157475' entity='mgr.x' 2024-04-26T20:00:01.167 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.198+0000 7f0be4400700 1 Processor -- start 2024-04-26T20:00:01.167 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.198+0000 7f0be4400700 1 -- start start 2024-04-26T20:00:01.167 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.198+0000 7f0be4400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc1077b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:01.167 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.199+0000 7f0be4400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc107d80 con 0x7f0bdc107390 2024-04-26T20:00:01.168 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.199+0000 7f0be1a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc1077b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:01.168 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.199+0000 7f0be1a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc1077b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60850/0 (socket says 172.21.15.94:60850) 2024-04-26T20:00:01.168 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.199+0000 7f0be1a00700 1 -- 172.21.15.94:0/1458896506 learned_addr learned my addr 172.21.15.94:0/1458896506 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:01.168 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.199+0000 7f0be1a00700 1 -- 172.21.15.94:0/1458896506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc108590 con 0x7f0bdc107390 2024-04-26T20:00:01.168 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.200+0000 7f0be1a00700 1 --2- 172.21.15.94:0/1458896506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc1077b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000bb40 tx=0x7f0bd000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acf0db413de655d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:01.168 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.200+0000 7f0bdbe00700 1 -- 172.21.15.94:0/1458896506 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd000dad0 con 0x7f0bdc107390 2024-04-26T20:00:01.168 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.200+0000 7f0bdbe00700 1 -- 172.21.15.94:0/1458896506 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bd00044e0 con 0x7f0bdc107390 2024-04-26T20:00:01.168 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.200+0000 7f0bdbe00700 1 -- 172.21.15.94:0/1458896506 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd0003da0 con 0x7f0bdc107390 2024-04-26T20:00:01.168 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.201+0000 7f0be4400700 1 -- 172.21.15.94:0/1458896506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 msgr2=0x7f0bdc1077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:01.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.201+0000 7f0be4400700 1 --2- 172.21.15.94:0/1458896506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc1077b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000bb40 tx=0x7f0bd000be50 comp rx=0 tx=0).stop 2024-04-26T20:00:01.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.201+0000 7f0be4400700 1 -- 172.21.15.94:0/1458896506 shutdown_connections 2024-04-26T20:00:01.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.201+0000 7f0be4400700 1 --2- 172.21.15.94:0/1458896506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc1077b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:01.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.201+0000 7f0be4400700 1 -- 172.21.15.94:0/1458896506 >> 172.21.15.94:0/1458896506 conn(0x7f0bdc102930 msgr2=0x7f0bdc104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:01.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.201+0000 7f0be4400700 1 -- 172.21.15.94:0/1458896506 shutdown_connections 2024-04-26T20:00:01.172 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.201+0000 7f0be4400700 1 -- 172.21.15.94:0/1458896506 wait complete. 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.202+0000 7f0be4400700 1 Processor -- start 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.203+0000 7f0be4400700 1 -- start start 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.203+0000 7f0be4400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.203+0000 7f0be4400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc108d00 con 0x7f0bdc107390 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.203+0000 7f0be1a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.203+0000 7f0be1a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc19ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60866/0 (socket says 172.21.15.94:60866) 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.203+0000 7f0be1a00700 1 -- 172.21.15.94:0/4148307677 learned_addr learned my addr 172.21.15.94:0/4148307677 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.204+0000 7f0be1a00700 1 -- 172.21.15.94:0/4148307677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc19f380 con 0x7f0bdc107390 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.204+0000 7f0be1a00700 1 --2- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc19ee40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0019040 tx=0x7f0bd0003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.204+0000 7f0bda000700 1 -- 172.21.15.94:0/4148307677 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd00174b0 con 0x7f0bdc107390 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.204+0000 7f0bda000700 1 -- 172.21.15.94:0/4148307677 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bd0024070 con 0x7f0bdc107390 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.204+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bdc19f610 con 0x7f0bdc107390 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.204+0000 7f0bda000700 1 -- 172.21.15.94:0/4148307677 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd001f850 con 0x7f0bdc107390 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.204+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bdc1a2300 con 0x7f0bdc107390 2024-04-26T20:00:01.173 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.206+0000 7f0bda000700 1 -- 172.21.15.94:0/4148307677 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0bd0017650 con 0x7f0bdc107390 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.206+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bdc064a80 con 0x7f0bdc107390 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.206+0000 7f0bda000700 1 --2- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f0bc4039600 0x7f0bc403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.206+0000 7f0bda000700 1 -- 172.21.15.94:0/4148307677 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0bd004da50 con 0x7f0bdc107390 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.207+0000 7f0be1000700 1 --2- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f0bc4039600 0x7f0bc403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.207+0000 7f0be1000700 1 --2- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f0bc4039600 0x7f0bc403bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8006fd0 tx=0x7f0bc8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.212+0000 7f0bda000700 1 -- 172.21.15.94:0/4148307677 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0bd0012070 con 0x7f0bdc107390 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.443+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f0bdc02e750 con 0x7f0bdc107390 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.446+0000 7f0bda000700 1 -- 172.21.15.94:0/4148307677 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f0bd0029e20 con 0x7f0bdc107390 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.449+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f0bc4039600 msgr2=0x7f0bc403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.449+0000 7f0be4400700 1 --2- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f0bc4039600 0x7f0bc403bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8006fd0 tx=0x7f0bc8006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.449+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 msgr2=0x7f0bdc19ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.449+0000 7f0be4400700 1 --2- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc19ee40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0019040 tx=0x7f0bd0003960 comp rx=0 tx=0).stop 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.449+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 shutdown_connections 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.449+0000 7f0be4400700 1 --2- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f0bc4039600 0x7f0bc403bac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:01.174 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.449+0000 7f0be4400700 1 --2- 172.21.15.94:0/4148307677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bdc107390 0x7f0bdc19ee40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:01.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.449+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 >> 172.21.15.94:0/4148307677 conn(0x7f0bdc102930 msgr2=0x7f0bdc195a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:01.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.450+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 shutdown_connections 2024-04-26T20:00:01.175 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:00.450+0000 7f0be4400700 1 -- 172.21.15.94:0/4148307677 wait complete. 2024-04-26T20:00:01.175 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling the dashboard module... 2024-04-26T20:00:01.702 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:01 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4148307677' entity='client.admin' 2024-04-26T20:00:01.702 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:01 smithi094 ceph-mon[28050]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:02.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:02 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/601447760' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T20:00:02.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: ignoring --setuser ceph since I am not root 2024-04-26T20:00:02.738 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: ignoring --setgroup ceph since I am not root 2024-04-26T20:00:03.154 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:02.777+0000 7f031e0fe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T20:00:03.154 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:03 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:03.022+0000 7f031e0fe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:00:03.154 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:03 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:03.152+0000 7f031e0fe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T20:00:03.254 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.766+0000 7f1fa0400700 1 Processor -- start 2024-04-26T20:00:03.260 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.766+0000 7f1fa0400700 1 -- start start 2024-04-26T20:00:03.260 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.766+0000 7f1fa0400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f9810bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:03.260 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.766+0000 7f1fa0400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9807cd50 con 0x7f1f98109870 2024-04-26T20:00:03.260 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.767+0000 7f1f9da00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f9810bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:03.260 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.767+0000 7f1f9da00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f9810bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44778/0 (socket says 172.21.15.94:44778) 2024-04-26T20:00:03.260 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.767+0000 7f1f9da00700 1 -- 172.21.15.94:0/2339456785 learned_addr learned my addr 172.21.15.94:0/2339456785 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:03.260 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.767+0000 7f1f9da00700 1 -- 172.21.15.94:0/2339456785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f9810c1a0 con 0x7f1f98109870 2024-04-26T20:00:03.260 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.768+0000 7f1f9da00700 1 --2- 172.21.15.94:0/2339456785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f9810bc60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0098a0 tx=0x7f1f8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d06074c79b2887 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.768+0000 7f1f97e00700 1 -- 172.21.15.94:0/2339456785 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f8c004160 con 0x7f1f98109870 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.768+0000 7f1f97e00700 1 -- 172.21.15.94:0/2339456785 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f8c004300 con 0x7f1f98109870 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.768+0000 7f1f97e00700 1 -- 172.21.15.94:0/2339456785 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f8c011a20 con 0x7f1f98109870 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.769+0000 7f1fa0400700 1 -- 172.21.15.94:0/2339456785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 msgr2=0x7f1f9810bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.769+0000 7f1fa0400700 1 --2- 172.21.15.94:0/2339456785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f9810bc60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0098a0 tx=0x7f1f8c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.769+0000 7f1fa0400700 1 -- 172.21.15.94:0/2339456785 shutdown_connections 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.769+0000 7f1fa0400700 1 --2- 172.21.15.94:0/2339456785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f9810bc60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.769+0000 7f1fa0400700 1 -- 172.21.15.94:0/2339456785 >> 172.21.15.94:0/2339456785 conn(0x7f1f981031e0 msgr2=0x7f1f98105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.769+0000 7f1fa0400700 1 -- 172.21.15.94:0/2339456785 shutdown_connections 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.769+0000 7f1fa0400700 1 -- 172.21.15.94:0/2339456785 wait complete. 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.770+0000 7f1fa0400700 1 Processor -- start 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.770+0000 7f1fa0400700 1 -- start start 2024-04-26T20:00:03.261 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.770+0000 7f1fa0400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f981090a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.770+0000 7f1fa0400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9810d0e0 con 0x7f1f98109870 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.771+0000 7f1f9da00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f981090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.771+0000 7f1f9da00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f981090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44792/0 (socket says 172.21.15.94:44792) 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.771+0000 7f1f9da00700 1 -- 172.21.15.94:0/601447760 learned_addr learned my addr 172.21.15.94:0/601447760 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.771+0000 7f1f9da00700 1 -- 172.21.15.94:0/601447760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f981095e0 con 0x7f1f98109870 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.771+0000 7f1f9da00700 1 --2- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f981090a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c004660 tx=0x7f1f8c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.772+0000 7f1f96000700 1 -- 172.21.15.94:0/601447760 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f8c003aa0 con 0x7f1f98109870 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.772+0000 7f1f96000700 1 -- 172.21.15.94:0/601447760 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f8c01b8f0 con 0x7f1f98109870 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.772+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f98107820 con 0x7f1f98109870 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.772+0000 7f1f96000700 1 -- 172.21.15.94:0/601447760 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f8c023900 con 0x7f1f98109870 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.772+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f98107d00 con 0x7f1f98109870 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.773+0000 7f1f96000700 1 -- 172.21.15.94:0/601447760 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1f8c023b70 con 0x7f1f98109870 2024-04-26T20:00:03.262 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.773+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f98064a80 con 0x7f1f98109870 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.773+0000 7f1f96000700 1 --2- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f1f80039600 0x7f1f8003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.773+0000 7f1f96000700 1 -- 172.21.15.94:0/601447760 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1f8c04de30 con 0x7f1f98109870 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.774+0000 7f1f9d000700 1 --2- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f1f80039600 0x7f1f8003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.774+0000 7f1f9d000700 1 --2- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f1f80039600 0x7f1f8003bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1f84006fd0 tx=0x7f1f84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:01.779+0000 7f1f96000700 1 -- 172.21.15.94:0/601447760 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1f8c01f030 con 0x7f1f98109870 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.074+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f1f981084a0 con 0x7f1f98109870 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.460+0000 7f1f96000700 1 -- 172.21.15.94:0/601447760 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f1f8c0292a0 con 0x7f1f98109870 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.464+0000 7f1f96000700 1 -- 172.21.15.94:0/601447760 <== mon.0 v2:172.21.15.94:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f1f8c02c3f0 con 0x7f1f98109870 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.466+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f1f80039600 msgr2=0x7f1f8003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.466+0000 7f1fa0400700 1 --2- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f1f80039600 0x7f1f8003bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1f84006fd0 tx=0x7f1f84006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.466+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 msgr2=0x7f1f981090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.466+0000 7f1fa0400700 1 --2- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f981090a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c004660 tx=0x7f1f8c011aa0 comp rx=0 tx=0).stop 2024-04-26T20:00:03.263 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.467+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 shutdown_connections 2024-04-26T20:00:03.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.467+0000 7f1fa0400700 1 --2- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7f1f80039600 0x7f1f8003bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:03.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.467+0000 7f1fa0400700 1 --2- 172.21.15.94:0/601447760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f98109870 0x7f1f981090a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:03.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.467+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 >> 172.21.15.94:0/601447760 conn(0x7f1f981031e0 msgr2=0x7f1f98103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:03.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.467+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 shutdown_connections 2024-04-26T20:00:03.264 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:02.467+0000 7f1fa0400700 1 -- 172.21.15.94:0/601447760 wait complete. 2024-04-26T20:00:03.686 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:03 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:03.564+0000 7f031e0fe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T20:00:03.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:03 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/601447760' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T20:00:03.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:03 smithi094 ceph-mon[28050]: mgrmap e9: x(active, since 23s) 2024-04-26T20:00:03.987 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:03 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:03.684+0000 7f031e0fe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T20:00:04.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:04 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/665498136' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T20:00:04.929 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:00:04.929 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T20:00:04.929 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T20:00:04.930 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T20:00:04.930 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T20:00:04.930 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:00:04.930 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.907+0000 7fb04fe00700 1 Processor -- start 2024-04-26T20:00:04.930 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.908+0000 7fb04fe00700 1 -- start start 2024-04-26T20:00:04.930 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.908+0000 7fb04fe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04807d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:04.931 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.908+0000 7fb04fe00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb04807d830 con 0x7fb04807ee00 2024-04-26T20:00:04.931 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.909+0000 7fb04d400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04807d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:04.931 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.909+0000 7fb04d400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04807d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44828/0 (socket says 172.21.15.94:44828) 2024-04-26T20:00:04.931 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.909+0000 7fb04d400700 1 -- 172.21.15.94:0/3815509993 learned_addr learned my addr 172.21.15.94:0/3815509993 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:04.931 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.909+0000 7fb04d400700 1 -- 172.21.15.94:0/3815509993 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04807d9b0 con 0x7fb04807ee00 2024-04-26T20:00:04.931 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.909+0000 7fb04d400700 1 --2- 172.21.15.94:0/3815509993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04807d260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0098a0 tx=0x7fb03c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58405d4f725c089a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.910+0000 7fb047e00700 1 -- 172.21.15.94:0/3815509993 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb03c004160 con 0x7fb04807ee00 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.910+0000 7fb047e00700 1 -- 172.21.15.94:0/3815509993 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb03c004300 con 0x7fb04807ee00 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.910+0000 7fb047e00700 1 -- 172.21.15.94:0/3815509993 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb03c011a20 con 0x7fb04807ee00 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.910+0000 7fb04fe00700 1 -- 172.21.15.94:0/3815509993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 msgr2=0x7fb04807d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.910+0000 7fb04fe00700 1 --2- 172.21.15.94:0/3815509993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04807d260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0098a0 tx=0x7fb03c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.911+0000 7fb04fe00700 1 -- 172.21.15.94:0/3815509993 shutdown_connections 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.911+0000 7fb04fe00700 1 --2- 172.21.15.94:0/3815509993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04807d260 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.911+0000 7fb04fe00700 1 -- 172.21.15.94:0/3815509993 >> 172.21.15.94:0/3815509993 conn(0x7fb0481042f0 msgr2=0x7fb048106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.911+0000 7fb04fe00700 1 -- 172.21.15.94:0/3815509993 shutdown_connections 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.911+0000 7fb04fe00700 1 -- 172.21.15.94:0/3815509993 wait complete. 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.912+0000 7fb04fe00700 1 Processor -- start 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.912+0000 7fb04fe00700 1 -- start start 2024-04-26T20:00:04.932 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.912+0000 7fb04fe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.912+0000 7fb04fe00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb04810f330 con 0x7fb04807ee00 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.912+0000 7fb04d400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.912+0000 7fb04d400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04819edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44830/0 (socket says 172.21.15.94:44830) 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.913+0000 7fb04d400700 1 -- 172.21.15.94:0/665498136 learned_addr learned my addr 172.21.15.94:0/665498136 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.913+0000 7fb04d400700 1 -- 172.21.15.94:0/665498136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04819f310 con 0x7fb04807ee00 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.913+0000 7fb04d400700 1 --2- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04819edd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb03c004660 tx=0x7fb03c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.913+0000 7fb046000700 1 -- 172.21.15.94:0/665498136 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb03c003aa0 con 0x7fb04807ee00 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.913+0000 7fb046000700 1 -- 172.21.15.94:0/665498136 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb03c01b8f0 con 0x7fb04807ee00 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.913+0000 7fb046000700 1 -- 172.21.15.94:0/665498136 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb03c023900 con 0x7fb04807ee00 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.913+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04819f5a0 con 0x7fb04807ee00 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.914+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0481a2290 con 0x7fb04807ee00 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.915+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb048064a80 con 0x7fb04807ee00 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.915+0000 7fb046000700 1 -- 172.21.15.94:0/665498136 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fb03c023bc0 con 0x7fb04807ee00 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.915+0000 7fb046000700 1 --2- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fb030039600 0x7fb03003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:04.933 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.916+0000 7fb046000700 1 -- 172.21.15.94:0/665498136 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb03c04dfa0 con 0x7fb04807ee00 2024-04-26T20:00:04.934 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.916+0000 7fb04ca00700 1 -- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fb030039600 msgr2=0x7fb03003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3651438916 2024-04-26T20:00:04.934 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.916+0000 7fb04ca00700 1 --2- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fb030039600 0x7fb03003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:00:04.934 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:03.921+0000 7fb046000700 1 -- 172.21.15.94:0/665498136 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb03c01f030 con 0x7fb04807ee00 2024-04-26T20:00:04.934 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.117+0000 7fb04ca00700 1 -- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fb030039600 msgr2=0x7fb03003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3651438916 2024-04-26T20:00:04.934 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.117+0000 7fb04ca00700 1 --2- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fb030039600 0x7fb03003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T20:00:04.934 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.222+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fb0481a2880 con 0x7fb04807ee00 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.223+0000 7fb046000700 1 -- 172.21.15.94:0/665498136 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fb03c01a3d0 con 0x7fb04807ee00 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.227+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fb030039600 msgr2=0x7fb03003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.227+0000 7fb04fe00700 1 --2- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fb030039600 0x7fb03003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.227+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 msgr2=0x7fb04819edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.227+0000 7fb04fe00700 1 --2- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04819edd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb03c004660 tx=0x7fb03c011aa0 comp rx=0 tx=0).stop 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.227+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 shutdown_connections 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.227+0000 7fb04fe00700 1 --2- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fb030039600 0x7fb03003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.227+0000 7fb04fe00700 1 --2- 172.21.15.94:0/665498136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04807ee00 0x7fb04819edd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.227+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 >> 172.21.15.94:0/665498136 conn(0x7fb0481042f0 msgr2=0x7fb048104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.228+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 shutdown_connections 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:04.228+0000 7fb04fe00700 1 -- 172.21.15.94:0/665498136 wait complete. 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for the mgr to restart... 2024-04-26T20:00:04.935 INFO:teuthology.orchestra.run.smithi094.stdout:Waiting for mgr epoch 9... 2024-04-26T20:00:05.551 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:05 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:05.255+0000 7f031e0fe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T20:00:05.552 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:05 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:05.494+0000 7f031e0fe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T20:00:05.987 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:05 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:05.712+0000 7f031e0fe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T20:00:06.463 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:06.088+0000 7f031e0fe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T20:00:06.463 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:06.210+0000 7f031e0fe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T20:00:06.713 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:06.461+0000 7f031e0fe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T20:00:06.964 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:06.712+0000 7f031e0fe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:00:07.987 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:07 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:07.508+0000 7f031e0fe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T20:00:08.451 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:08 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:08.056+0000 7f031e0fe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T20:00:08.451 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:08 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:08.185+0000 7f031e0fe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T20:00:08.451 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:08 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:08.449+0000 7f031e0fe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T20:00:09.487 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:09.089+0000 7f031e0fe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T20:00:09.487 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:09.212+0000 7f031e0fe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T20:00:09.843 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:09.536+0000 7f031e0fe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T20:00:09.843 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:09.714+0000 7f031e0fe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T20:00:09.843 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:09.841+0000 7f031e0fe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T20:00:10.237 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:09.989+0000 7f031e0fe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T20:00:10.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:10.374+0000 7f031e0fe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T20:00:11.411 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:11.119+0000 7f031e0fe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:00:11.409+0000 7f031e0fe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: Active manager daemon x restarted 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: Activating manager daemon x 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: mgrmap e10: x(active, starting, since 0.00882503s) 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T20:00:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T20:00:11.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:11 smithi094 ceph-mon[28050]: Manager daemon x is now available 2024-04-26T20:00:12.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:00:12.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T20:00:12.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T20:00:12.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:12 smithi094 ceph-mon[28050]: mgrmap e11: x(active, since 1.00862s) 2024-04-26T20:00:13.090 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:00:13.090 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T20:00:13.090 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T20:00:13.090 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:00:13.090 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.550+0000 7fa242e00700 1 Processor -- start 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.550+0000 7fa242e00700 1 -- start start 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.550+0000 7fa242e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c07d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.550+0000 7fa242e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa23c07d830 con 0x7fa23c07ee00 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.551+0000 7fa23be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c07d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.551+0000 7fa23be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c07d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44846/0 (socket says 172.21.15.94:44846) 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.551+0000 7fa23be00700 1 -- 172.21.15.94:0/2604679767 learned_addr learned my addr 172.21.15.94:0/2604679767 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.551+0000 7fa23be00700 1 -- 172.21.15.94:0/2604679767 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23c07d9b0 con 0x7fa23c07ee00 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.552+0000 7fa23be00700 1 --2- 172.21.15.94:0/2604679767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c07d260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa22c0098a0 tx=0x7fa22c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78cb9b771b94e99 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.552+0000 7fa23b400700 1 -- 172.21.15.94:0/2604679767 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa22c004160 con 0x7fa23c07ee00 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.552+0000 7fa23b400700 1 -- 172.21.15.94:0/2604679767 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa22c004300 con 0x7fa23c07ee00 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.552+0000 7fa23b400700 1 -- 172.21.15.94:0/2604679767 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa22c011a20 con 0x7fa23c07ee00 2024-04-26T20:00:13.092 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.553+0000 7fa242e00700 1 -- 172.21.15.94:0/2604679767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 msgr2=0x7fa23c07d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:13.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.553+0000 7fa242e00700 1 --2- 172.21.15.94:0/2604679767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c07d260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa22c0098a0 tx=0x7fa22c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:00:13.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.553+0000 7fa242e00700 1 -- 172.21.15.94:0/2604679767 shutdown_connections 2024-04-26T20:00:13.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.553+0000 7fa242e00700 1 --2- 172.21.15.94:0/2604679767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c07d260 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:13.095 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.553+0000 7fa242e00700 1 -- 172.21.15.94:0/2604679767 >> 172.21.15.94:0/2604679767 conn(0x7fa23c1042f0 msgr2=0x7fa23c106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.553+0000 7fa242e00700 1 -- 172.21.15.94:0/2604679767 shutdown_connections 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.553+0000 7fa242e00700 1 -- 172.21.15.94:0/2604679767 wait complete. 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.554+0000 7fa242e00700 1 Processor -- start 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.554+0000 7fa242e00700 1 -- start start 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.555+0000 7fa242e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.555+0000 7fa242e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa23c10f330 con 0x7fa23c07ee00 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.555+0000 7fa23be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.555+0000 7fa23be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44852/0 (socket says 172.21.15.94:44852) 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.555+0000 7fa23be00700 1 -- 172.21.15.94:0/6154373 learned_addr learned my addr 172.21.15.94:0/6154373 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.555+0000 7fa23be00700 1 -- 172.21.15.94:0/6154373 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23c19f310 con 0x7fa23c07ee00 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.556+0000 7fa23be00700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c19edd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa22c004660 tx=0x7fa22c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.556+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa22c003aa0 con 0x7fa23c07ee00 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.556+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa22c01b8f0 con 0x7fa23c07ee00 2024-04-26T20:00:13.096 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.556+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa22c023900 con 0x7fa23c07ee00 2024-04-26T20:00:13.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.556+0000 7fa242e00700 1 -- 172.21.15.94:0/6154373 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa23c19f5a0 con 0x7fa23c07ee00 2024-04-26T20:00:13.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.556+0000 7fa242e00700 1 -- 172.21.15.94:0/6154373 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa23c1a2290 con 0x7fa23c07ee00 2024-04-26T20:00:13.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.557+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fa22c01baa0 con 0x7fa23c07ee00 2024-04-26T20:00:13.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.558+0000 7fa239600700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 0x7fa22003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:13.097 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.558+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 --> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa22003c1d0 con 0x7fa220039600 2024-04-26T20:00:13.100 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.558+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa22c04dfe0 con 0x7fa23c07ee00 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.558+0000 7fa233400700 1 -- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 msgr2=0x7fa22003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3651438916 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.558+0000 7fa233400700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 0x7fa22003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.759+0000 7fa233400700 1 -- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 msgr2=0x7fa22003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3651438916 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:05.759+0000 7fa233400700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 0x7fa22003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:06.160+0000 7fa233400700 1 -- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 msgr2=0x7fa22003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3651438916 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:06.160+0000 7fa233400700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 0x7fa22003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:06.961+0000 7fa233400700 1 -- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 msgr2=0x7fa22003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3651438916 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:06.961+0000 7fa233400700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 0x7fa22003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:08.563+0000 7fa233400700 1 -- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 msgr2=0x7fa22003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.94:6800/3651438916 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:08.563+0000 7fa233400700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 0x7fa22003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:11.418+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7fa22c00b3e0 con 0x7fa23c07ee00 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:11.418+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 msgr2=0x7fa22003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:00:13.101 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:11.418+0000 7fa239600700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 0x7fa22003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.418+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7fa22c04fc50 con 0x7fa23c07ee00 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.419+0000 7fa239600700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa22003cfe0 0x7fa22003f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.419+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa22003c1d0 con 0x7fa22003cfe0 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.419+0000 7fa233400700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa22003cfe0 0x7fa22003f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.421+0000 7fa233400700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa22003cfe0 0x7fa22003f3d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa224003a10 tx=0x7fa2240092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.422+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fa22003c1d0 con 0x7fa22003cfe0 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.431+0000 7fa242e00700 1 -- 172.21.15.94:0/6154373 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fa23c064a80 con 0x7fa22003cfe0 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.431+0000 7fa239600700 1 -- 172.21.15.94:0/6154373 <== mgr.14152 v2:172.21.15.94:6800/2112565670 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fa23c064a80 con 0x7fa22003cfe0 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 -- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa22003cfe0 msgr2=0x7fa22003f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa22003cfe0 0x7fa22003f3d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa224003a10 tx=0x7fa2240092b0 comp rx=0 tx=0).stop 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 -- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 msgr2=0x7fa23c19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c19edd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa22c004660 tx=0x7fa22c011aa0 comp rx=0 tx=0).stop 2024-04-26T20:00:13.102 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 -- 172.21.15.94:0/6154373 shutdown_connections 2024-04-26T20:00:13.103 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa22003cfe0 0x7fa22003f3d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:13.103 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa23c07ee00 0x7fa23c19edd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:13.103 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 --2- 172.21.15.94:0/6154373 >> [v2:172.21.15.94:6800/3651438916,v1:172.21.15.94:6801/3651438916] conn(0x7fa220039600 0x7fa22003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:13.103 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 -- 172.21.15.94:0/6154373 >> 172.21.15.94:0/6154373 conn(0x7fa23c1042f0 msgr2=0x7fa23c104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:13.103 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 -- 172.21.15.94:0/6154373 shutdown_connections 2024-04-26T20:00:13.103 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:12.432+0000 7fa242e00700 1 -- 172.21.15.94:0/6154373 wait complete. 2024-04-26T20:00:13.103 INFO:teuthology.orchestra.run.smithi094.stdout:mgr epoch 9 is available 2024-04-26T20:00:13.103 INFO:teuthology.orchestra.run.smithi094.stdout:Generating a dashboard self-signed certificate... 2024-04-26T20:00:13.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:13 smithi094 ceph-mon[28050]: [26/Apr/2024:20:00:11] ENGINE Bus STARTING 2024-04-26T20:00:13.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:13 smithi094 ceph-mon[28050]: [26/Apr/2024:20:00:12] ENGINE Serving on http://172.21.15.94:8765 2024-04-26T20:00:13.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:13 smithi094 ceph-mon[28050]: [26/Apr/2024:20:00:12] ENGINE Serving on https://172.21.15.94:7150 2024-04-26T20:00:13.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:13 smithi094 ceph-mon[28050]: [26/Apr/2024:20:00:12] ENGINE Bus STARTED 2024-04-26T20:00:13.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:13 smithi094 ceph-mon[28050]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T20:00:13.717 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:13 smithi094 ceph-mon[28050]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T20:00:14.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:14 smithi094 ceph-mon[28050]: mgrmap e12: x(active, since 2s) 2024-04-26T20:00:14.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:15.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T20:00:15.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.287+0000 7f0919400700 1 Processor -- start 2024-04-26T20:00:15.319 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.287+0000 7f0919400700 1 -- start start 2024-04-26T20:00:15.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.287+0000 7f0919400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091410bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:15.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.288+0000 7f0919400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f091407cdb0 con 0x7f0914109810 2024-04-26T20:00:15.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.288+0000 7f0912800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091410bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:15.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.288+0000 7f0912800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091410bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47714/0 (socket says 172.21.15.94:47714) 2024-04-26T20:00:15.320 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.288+0000 7f0912800700 1 -- 172.21.15.94:0/1421458912 learned_addr learned my addr 172.21.15.94:0/1421458912 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:15.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.289+0000 7f0912800700 1 -- 172.21.15.94:0/1421458912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091410c140 con 0x7f0914109810 2024-04-26T20:00:15.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.289+0000 7f0912800700 1 --2- 172.21.15.94:0/1421458912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091410bc00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f09040098a0 tx=0x7f0904009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c869710de5005d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:15.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.289+0000 7f0911400700 1 -- 172.21.15.94:0/1421458912 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0904004160 con 0x7f0914109810 2024-04-26T20:00:15.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.290+0000 7f0911400700 1 -- 172.21.15.94:0/1421458912 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0904004300 con 0x7f0914109810 2024-04-26T20:00:15.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.290+0000 7f0911400700 1 -- 172.21.15.94:0/1421458912 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0904011a20 con 0x7f0914109810 2024-04-26T20:00:15.321 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.290+0000 7f0919400700 1 -- 172.21.15.94:0/1421458912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 msgr2=0x7f091410bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:15.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.290+0000 7f0919400700 1 --2- 172.21.15.94:0/1421458912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091410bc00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f09040098a0 tx=0x7f0904009bb0 comp rx=0 tx=0).stop 2024-04-26T20:00:15.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.291+0000 7f0919400700 1 -- 172.21.15.94:0/1421458912 shutdown_connections 2024-04-26T20:00:15.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.291+0000 7f0919400700 1 --2- 172.21.15.94:0/1421458912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091410bc00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:15.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.291+0000 7f0919400700 1 -- 172.21.15.94:0/1421458912 >> 172.21.15.94:0/1421458912 conn(0x7f09141031e0 msgr2=0x7f0914105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:15.322 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.291+0000 7f0919400700 1 -- 172.21.15.94:0/1421458912 shutdown_connections 2024-04-26T20:00:15.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.291+0000 7f0919400700 1 -- 172.21.15.94:0/1421458912 wait complete. 2024-04-26T20:00:15.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.292+0000 7f0919400700 1 Processor -- start 2024-04-26T20:00:15.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.292+0000 7f0919400700 1 -- start start 2024-04-26T20:00:15.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.292+0000 7f0919400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:15.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.292+0000 7f0919400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f091410d0e0 con 0x7f0914109810 2024-04-26T20:00:15.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.293+0000 7f0912800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:15.323 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.293+0000 7f0912800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47728/0 (socket says 172.21.15.94:47728) 2024-04-26T20:00:15.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.293+0000 7f0912800700 1 -- 172.21.15.94:0/1979862707 learned_addr learned my addr 172.21.15.94:0/1979862707 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:15.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.293+0000 7f0912800700 1 -- 172.21.15.94:0/1979862707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091419f3a0 con 0x7f0914109810 2024-04-26T20:00:15.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.293+0000 7f0912800700 1 --2- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091419ee60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0904004660 tx=0x7f0904011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:15.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.294+0000 7f090b400700 1 -- 172.21.15.94:0/1979862707 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0904003aa0 con 0x7f0914109810 2024-04-26T20:00:15.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.294+0000 7f090b400700 1 -- 172.21.15.94:0/1979862707 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f090401b8f0 con 0x7f0914109810 2024-04-26T20:00:15.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.294+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f091419f630 con 0x7f0914109810 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.294+0000 7f090b400700 1 -- 172.21.15.94:0/1979862707 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0904023900 con 0x7f0914109810 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.294+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09141a2320 con 0x7f0914109810 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.295+0000 7f090b400700 1 -- 172.21.15.94:0/1979862707 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f0904023b70 con 0x7f0914109810 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.295+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0914064a80 con 0x7f0914109810 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.295+0000 7f090b400700 1 --2- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f08f8039660 0x7f08f803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.296+0000 7f090b400700 1 -- 172.21.15.94:0/1979862707 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f090404dff0 con 0x7f0914109810 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.296+0000 7f0911e00700 1 --2- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f08f8039660 0x7f08f803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.296+0000 7f0911e00700 1 --2- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f08f8039660 0x7f08f803bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0900006fd0 tx=0x7f0900006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.303+0000 7f090b400700 1 -- 172.21.15.94:0/1979862707 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f090401f030 con 0x7f0914109810 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.542+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f09141042e0 con 0x7f08f8039660 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.690+0000 7f090b400700 1 -- 172.21.15.94:0/1979862707 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f09141042e0 con 0x7f08f8039660 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.693+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f08f8039660 msgr2=0x7f08f803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.693+0000 7f0919400700 1 --2- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f08f8039660 0x7f08f803bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0900006fd0 tx=0x7f0900006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:15.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.693+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 msgr2=0x7f091419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:15.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.693+0000 7f0919400700 1 --2- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091419ee60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0904004660 tx=0x7f0904011aa0 comp rx=0 tx=0).stop 2024-04-26T20:00:15.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.693+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 shutdown_connections 2024-04-26T20:00:15.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.693+0000 7f0919400700 1 --2- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f08f8039660 0x7f08f803bb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:15.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.693+0000 7f0919400700 1 --2- 172.21.15.94:0/1979862707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0914109810 0x7f091419ee60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:15.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.693+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 >> 172.21.15.94:0/1979862707 conn(0x7f09141031e0 msgr2=0x7f0914103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:15.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.694+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 shutdown_connections 2024-04-26T20:00:15.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:14.694+0000 7f0919400700 1 -- 172.21.15.94:0/1979862707 wait complete. 2024-04-26T20:00:15.326 INFO:teuthology.orchestra.run.smithi094.stdout:Creating initial admin user... 2024-04-26T20:00:15.646 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:15 smithi094 ceph-mon[28050]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:00:15.646 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:15.646 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:17.318 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$PZv6DYxocbFrWi3sPVBBjeJbpgAjUYymIgIoZMk4Qg4FrJBTaY5IK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714161616, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T20:00:17.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.103+0000 7f7472e00700 1 Processor -- start 2024-04-26T20:00:17.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.103+0000 7f7472e00700 1 -- start start 2024-04-26T20:00:17.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.104+0000 7f7472e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c1079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:17.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.104+0000 7f7472e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f746c07ce30 con 0x7f746c107580 2024-04-26T20:00:17.324 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.105+0000 7f746be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c1079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.105+0000 7f746be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c1079a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47744/0 (socket says 172.21.15.94:47744) 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.105+0000 7f746be00700 1 -- 172.21.15.94:0/460337842 learned_addr learned my addr 172.21.15.94:0/460337842 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.105+0000 7f746be00700 1 -- 172.21.15.94:0/460337842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c107ee0 con 0x7f746c107580 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.106+0000 7f746be00700 1 --2- 172.21.15.94:0/460337842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c1079a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f745c0098a0 tx=0x7f745c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc45c1309e2b581 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.106+0000 7f746aa00700 1 -- 172.21.15.94:0/460337842 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f745c004160 con 0x7f746c107580 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.106+0000 7f746aa00700 1 -- 172.21.15.94:0/460337842 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f745c004300 con 0x7f746c107580 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.106+0000 7f746aa00700 1 -- 172.21.15.94:0/460337842 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f745c011a20 con 0x7f746c107580 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.107+0000 7f7472e00700 1 -- 172.21.15.94:0/460337842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 msgr2=0x7f746c1079a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.107+0000 7f7472e00700 1 --2- 172.21.15.94:0/460337842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c1079a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f745c0098a0 tx=0x7f745c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.107+0000 7f7472e00700 1 -- 172.21.15.94:0/460337842 shutdown_connections 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.107+0000 7f7472e00700 1 --2- 172.21.15.94:0/460337842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c1079a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:17.325 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.107+0000 7f7472e00700 1 -- 172.21.15.94:0/460337842 >> 172.21.15.94:0/460337842 conn(0x7f746c1031e0 msgr2=0x7f746c105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.107+0000 7f7472e00700 1 -- 172.21.15.94:0/460337842 shutdown_connections 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.107+0000 7f7472e00700 1 -- 172.21.15.94:0/460337842 wait complete. 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.108+0000 7f7472e00700 1 Processor -- start 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.109+0000 7f7472e00700 1 -- start start 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.109+0000 7f7472e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.109+0000 7f7472e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f746c108f50 con 0x7f746c107580 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.109+0000 7f746be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.109+0000 7f746be00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47758/0 (socket says 172.21.15.94:47758) 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.109+0000 7f746be00700 1 -- 172.21.15.94:0/3436203867 learned_addr learned my addr 172.21.15.94:0/3436203867 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.110+0000 7f746be00700 1 -- 172.21.15.94:0/3436203867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c19f310 con 0x7f746c107580 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.110+0000 7f746be00700 1 --2- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c19edd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f745c004660 tx=0x7f745c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:17.326 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.110+0000 7f7468c00700 1 -- 172.21.15.94:0/3436203867 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f745c003aa0 con 0x7f746c107580 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.110+0000 7f7468c00700 1 -- 172.21.15.94:0/3436203867 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f745c01b8f0 con 0x7f746c107580 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.110+0000 7f7468c00700 1 -- 172.21.15.94:0/3436203867 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f745c023900 con 0x7f746c107580 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.110+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746c19f5a0 con 0x7f746c107580 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.111+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f746c1a2290 con 0x7f746c107580 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.112+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746c064a80 con 0x7f746c107580 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.112+0000 7f7468c00700 1 -- 172.21.15.94:0/3436203867 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f745c01baa0 con 0x7f746c107580 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.113+0000 7f7468c00700 1 --2- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7450039600 0x7f745003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.113+0000 7f7468c00700 1 -- 172.21.15.94:0/3436203867 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f745c04e110 con 0x7f746c107580 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.113+0000 7f746b400700 1 --2- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7450039600 0x7f745003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.114+0000 7f746b400700 1 --2- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7450039600 0x7f745003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7458006fd0 tx=0x7f7458006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.119+0000 7f7468c00700 1 -- 172.21.15.94:0/3436203867 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f745c01f030 con 0x7f746c107580 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.380+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f746c02e6d0 con 0x7f7450039600 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.630+0000 7f7468c00700 1 -- 172.21.15.94:0/3436203867 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f746c02e6d0 con 0x7f7450039600 2024-04-26T20:00:17.327 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.632+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7450039600 msgr2=0x7f745003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.632+0000 7f7472e00700 1 --2- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7450039600 0x7f745003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7458006fd0 tx=0x7f7458006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.632+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 msgr2=0x7f746c19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.632+0000 7f7472e00700 1 --2- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c19edd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f745c004660 tx=0x7f745c011aa0 comp rx=0 tx=0).stop 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.633+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 shutdown_connections 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.633+0000 7f7472e00700 1 --2- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7450039600 0x7f745003bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.633+0000 7f7472e00700 1 --2- 172.21.15.94:0/3436203867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f746c107580 0x7f746c19edd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.633+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 >> 172.21.15.94:0/3436203867 conn(0x7f746c1031e0 msgr2=0x7f746c1959f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.633+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 shutdown_connections 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:16.633+0000 7f7472e00700 1 -- 172.21.15.94:0/3436203867 wait complete. 2024-04-26T20:00:17.328 INFO:teuthology.orchestra.run.smithi094.stdout:Fetching dashboard port number... 2024-04-26T20:00:17.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:17.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:17 smithi094 ceph-mon[28050]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:00:17.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:19.593 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:19 smithi094 ceph-mon[28050]: mgrmap e13: x(active, since 6s) 2024-04-26T20:00:19.593 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:19 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1223874323' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T20:00:19.602 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T20:00:19.603 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.122+0000 7f07fa000700 1 Processor -- start 2024-04-26T20:00:19.604 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.123+0000 7f07fa000700 1 -- start start 2024-04-26T20:00:19.604 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.123+0000 7f07fa000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f407d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:19.604 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.123+0000 7f07fa000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f07f407d870 con 0x7f07f407ee40 2024-04-26T20:00:19.604 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.123+0000 7f07f3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f407d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:19.604 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.123+0000 7f07f3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f407d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47766/0 (socket says 172.21.15.94:47766) 2024-04-26T20:00:19.604 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.123+0000 7f07f3400700 1 -- 172.21.15.94:0/1967027500 learned_addr learned my addr 172.21.15.94:0/1967027500 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:19.604 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.124+0000 7f07f3400700 1 -- 172.21.15.94:0/1967027500 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f407d9f0 con 0x7f07f407ee40 2024-04-26T20:00:19.604 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.124+0000 7f07f3400700 1 --2- 172.21.15.94:0/1967027500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f407d2a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f07e40098a0 tx=0x7f07e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c460f7506298c70c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:19.609 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.124+0000 7f07f2000700 1 -- 172.21.15.94:0/1967027500 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e4004160 con 0x7f07f407ee40 2024-04-26T20:00:19.609 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.124+0000 7f07f2000700 1 -- 172.21.15.94:0/1967027500 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07e4004300 con 0x7f07f407ee40 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.124+0000 7f07f2000700 1 -- 172.21.15.94:0/1967027500 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e4011a20 con 0x7f07f407ee40 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.125+0000 7f07fa000700 1 -- 172.21.15.94:0/1967027500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 msgr2=0x7f07f407d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.125+0000 7f07fa000700 1 --2- 172.21.15.94:0/1967027500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f407d2a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f07e40098a0 tx=0x7f07e4009bb0 comp rx=0 tx=0).stop 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.125+0000 7f07fa000700 1 -- 172.21.15.94:0/1967027500 shutdown_connections 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.125+0000 7f07fa000700 1 --2- 172.21.15.94:0/1967027500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f407d2a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.125+0000 7f07fa000700 1 -- 172.21.15.94:0/1967027500 >> 172.21.15.94:0/1967027500 conn(0x7f07f41042f0 msgr2=0x7f07f4106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.125+0000 7f07fa000700 1 -- 172.21.15.94:0/1967027500 shutdown_connections 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.125+0000 7f07fa000700 1 -- 172.21.15.94:0/1967027500 wait complete. 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.126+0000 7f07fa000700 1 Processor -- start 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.126+0000 7f07fa000700 1 -- start start 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.127+0000 7f07fa000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.127+0000 7f07fa000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f07f410f2f0 con 0x7f07f407ee40 2024-04-26T20:00:19.610 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.127+0000 7f07f3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.127+0000 7f07f3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f419ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47772/0 (socket says 172.21.15.94:47772) 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.127+0000 7f07f3400700 1 -- 172.21.15.94:0/1223874323 learned_addr learned my addr 172.21.15.94:0/1223874323 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.127+0000 7f07f3400700 1 -- 172.21.15.94:0/1223874323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f419f360 con 0x7f07f407ee40 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.127+0000 7f07f3400700 1 --2- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f419ee20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f07e4000c00 tx=0x7f07e4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.128+0000 7f07ebe00700 1 -- 172.21.15.94:0/1223874323 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e4003aa0 con 0x7f07f407ee40 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.128+0000 7f07ebe00700 1 -- 172.21.15.94:0/1223874323 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07e401b8f0 con 0x7f07f407ee40 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.128+0000 7f07ebe00700 1 -- 172.21.15.94:0/1223874323 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e4023890 con 0x7f07f407ee40 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.128+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f419f5f0 con 0x7f07f407ee40 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.128+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f41a22e0 con 0x7f07f407ee40 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.129+0000 7f07ebe00700 1 -- 172.21.15.94:0/1223874323 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f07e401b440 con 0x7f07f407ee40 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.129+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f4064a80 con 0x7f07f407ee40 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.129+0000 7f07ebe00700 1 --2- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f07d8039660 0x7f07d803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.130+0000 7f07ebe00700 1 -- 172.21.15.94:0/1223874323 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f07e404e030 con 0x7f07f407ee40 2024-04-26T20:00:19.611 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.130+0000 7f07f2a00700 1 --2- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f07d8039660 0x7f07d803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.130+0000 7f07f2a00700 1 --2- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f07d8039660 0x7f07d803bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f07e0006fd0 tx=0x7f07e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.135+0000 7f07ebe00700 1 -- 172.21.15.94:0/1223874323 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f07e401f030 con 0x7f07f407ee40 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.325+0000 7f07ebe00700 1 -- 172.21.15.94:0/1223874323 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f07e402c770 con 0x7f07f407ee40 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.362+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f07f41a28d0 con 0x7f07f407ee40 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.363+0000 7f07ebe00700 1 -- 172.21.15.94:0/1223874323 <== mon.0 v2:172.21.15.94:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f07e4021300 con 0x7f07f407ee40 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f07d8039660 msgr2=0x7f07d803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 --2- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f07d8039660 0x7f07d803bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f07e0006fd0 tx=0x7f07e0006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 msgr2=0x7f07f419ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 --2- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f419ee20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f07e4000c00 tx=0x7f07e4011aa0 comp rx=0 tx=0).stop 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 shutdown_connections 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 --2- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f07d8039660 0x7f07d803bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 --2- 172.21.15.94:0/1223874323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f407ee40 0x7f07f419ee20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:19.612 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 >> 172.21.15.94:0/1223874323 conn(0x7f07f41042f0 msgr2=0x7f07f4104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:19.613 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 shutdown_connections 2024-04-26T20:00:19.613 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:18.366+0000 7f07fa000700 1 -- 172.21.15.94:0/1223874323 wait complete. 2024-04-26T20:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T20:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout disabled 2024-04-26T20:00:19.665 INFO:teuthology.orchestra.run.smithi094.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T20:00:19.665 INFO:teuthology.orchestra.run.smithi094.stdout:systemctl: stdout inactive 2024-04-26T20:00:19.665 INFO:teuthology.orchestra.run.smithi094.stdout:firewalld.service is not enabled 2024-04-26T20:00:19.666 INFO:teuthology.orchestra.run.smithi094.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T20:00:19.673 INFO:teuthology.orchestra.run.smithi094.stdout:Ceph Dashboard is now available at: 2024-04-26T20:00:19.673 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:19.673 INFO:teuthology.orchestra.run.smithi094.stdout: URL: https://smithi094.front.sepia.ceph.com:8443/ 2024-04-26T20:00:19.673 INFO:teuthology.orchestra.run.smithi094.stdout: User: admin 2024-04-26T20:00:19.673 INFO:teuthology.orchestra.run.smithi094.stdout: Password: 4p4cd88rwh 2024-04-26T20:00:19.673 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:19.674 INFO:teuthology.orchestra.run.smithi094.stdout:Saving cluster configuration to /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config directory 2024-04-26T20:00:19.676 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling autotune for osd_memory_target 2024-04-26T20:00:21.149 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.284+0000 7fc70e200700 1 Processor -- start 2024-04-26T20:00:21.149 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.284+0000 7fc70e200700 1 -- start start 2024-04-26T20:00:21.149 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.284+0000 7fc70e200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc708109180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:21.149 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.284+0000 7fc70e200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc708109750 con 0x7fc708108d60 2024-04-26T20:00:21.149 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.285+0000 7fc707400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc708109180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:21.149 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.285+0000 7fc707400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc708109180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47778/0 (socket says 172.21.15.94:47778) 2024-04-26T20:00:21.149 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.285+0000 7fc707400700 1 -- 172.21.15.94:0/3624812341 learned_addr learned my addr 172.21.15.94:0/3624812341 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:21.149 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.285+0000 7fc707400700 1 -- 172.21.15.94:0/3624812341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc708109f60 con 0x7fc708108d60 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.286+0000 7fc707400700 1 --2- 172.21.15.94:0/3624812341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc708109180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80098a0 tx=0x7fc6f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b73b7b0a6a8e43 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.286+0000 7fc706000700 1 -- 172.21.15.94:0/3624812341 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f8004160 con 0x7fc708108d60 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.286+0000 7fc706000700 1 -- 172.21.15.94:0/3624812341 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6f8004300 con 0x7fc708108d60 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.286+0000 7fc706000700 1 -- 172.21.15.94:0/3624812341 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f8011a20 con 0x7fc708108d60 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.287+0000 7fc70e200700 1 -- 172.21.15.94:0/3624812341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 msgr2=0x7fc708109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.287+0000 7fc70e200700 1 --2- 172.21.15.94:0/3624812341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc708109180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc6f80098a0 tx=0x7fc6f8009bb0 comp rx=0 tx=0).stop 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.287+0000 7fc70e200700 1 -- 172.21.15.94:0/3624812341 shutdown_connections 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.287+0000 7fc70e200700 1 --2- 172.21.15.94:0/3624812341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc708109180 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.287+0000 7fc70e200700 1 -- 172.21.15.94:0/3624812341 >> 172.21.15.94:0/3624812341 conn(0x7fc708104320 msgr2=0x7fc708106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.287+0000 7fc70e200700 1 -- 172.21.15.94:0/3624812341 shutdown_connections 2024-04-26T20:00:21.150 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.287+0000 7fc70e200700 1 -- 172.21.15.94:0/3624812341 wait complete. 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.288+0000 7fc70e200700 1 Processor -- start 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.288+0000 7fc70e200700 1 -- start start 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.288+0000 7fc70e200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc70819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.288+0000 7fc70e200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc70810d310 con 0x7fc708108d60 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.289+0000 7fc707400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc70819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.289+0000 7fc707400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc70819eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47786/0 (socket says 172.21.15.94:47786) 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.289+0000 7fc707400700 1 -- 172.21.15.94:0/1563695588 learned_addr learned my addr 172.21.15.94:0/1563695588 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.289+0000 7fc707400700 1 -- 172.21.15.94:0/1563695588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70819f3f0 con 0x7fc708108d60 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.289+0000 7fc707400700 1 --2- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc70819eeb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8000c00 tx=0x7fc6f8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.290+0000 7fc6ffe00700 1 -- 172.21.15.94:0/1563695588 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f8003aa0 con 0x7fc708108d60 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.290+0000 7fc6ffe00700 1 -- 172.21.15.94:0/1563695588 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6f801b8f0 con 0x7fc708108d60 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.290+0000 7fc6ffe00700 1 -- 172.21.15.94:0/1563695588 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6f8023890 con 0x7fc708108d60 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.290+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc70819f680 con 0x7fc708108d60 2024-04-26T20:00:21.151 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.290+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7081a2320 con 0x7fc708108d60 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.291+0000 7fc6ffe00700 1 -- 172.21.15.94:0/1563695588 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc6f801b440 con 0x7fc708108d60 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.291+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc70810d490 con 0x7fc708108d60 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.291+0000 7fc6ffe00700 1 --2- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc6ec039700 0x7fc6ec03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.292+0000 7fc6ffe00700 1 -- 172.21.15.94:0/1563695588 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc6f804e110 con 0x7fc708108d60 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.292+0000 7fc706a00700 1 --2- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc6ec039700 0x7fc6ec03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.292+0000 7fc706a00700 1 --2- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc6ec039700 0x7fc6ec03bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4006fd0 tx=0x7fc6f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.298+0000 7fc6ffe00700 1 -- 172.21.15.94:0/1563695588 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc6f801f030 con 0x7fc708108d60 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.521+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fc7081a2990 con 0x7fc708108d60 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.522+0000 7fc6ffe00700 1 -- 172.21.15.94:0/1563695588 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fc6f80189c0 con 0x7fc708108d60 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc6ec039700 msgr2=0x7fc6ec03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 --2- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc6ec039700 0x7fc6ec03bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4006fd0 tx=0x7fc6f4006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 msgr2=0x7fc70819eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:21.152 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 --2- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc70819eeb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8000c00 tx=0x7fc6f8011aa0 comp rx=0 tx=0).stop 2024-04-26T20:00:21.153 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 shutdown_connections 2024-04-26T20:00:21.153 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 --2- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc6ec039700 0x7fc6ec03bbc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:21.153 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 --2- 172.21.15.94:0/1563695588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc708108d60 0x7fc70819eeb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:21.153 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 >> 172.21.15.94:0/1563695588 conn(0x7fc708104320 msgr2=0x7fc708104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:21.153 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 shutdown_connections 2024-04-26T20:00:21.153 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:20.525+0000 7fc70e200700 1 -- 172.21.15.94:0/1563695588 wait complete. 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.773+0000 7efd0c000700 1 Processor -- start 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.773+0000 7efd0c000700 1 -- start start 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.774+0000 7efd0c000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd04112040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.774+0000 7efd0c000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efd04112580 con 0x7efd0410fc50 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.774+0000 7efd09600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd04112040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.774+0000 7efd09600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd04112040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51550/0 (socket says 172.21.15.94:51550) 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.774+0000 7efd09600700 1 -- 172.21.15.94:0/3580040473 learned_addr learned my addr 172.21.15.94:0/3580040473 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.775+0000 7efd09600700 1 -- 172.21.15.94:0/3580040473 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd04112700 con 0x7efd0410fc50 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.775+0000 7efd09600700 1 --2- 172.21.15.94:0/3580040473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd04112040 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efcf80098a0 tx=0x7efcf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49db5f369b998 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.775+0000 7efd03e00700 1 -- 172.21.15.94:0/3580040473 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf8004160 con 0x7efd0410fc50 2024-04-26T20:00:22.843 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.775+0000 7efd03e00700 1 -- 172.21.15.94:0/3580040473 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcf8004300 con 0x7efd0410fc50 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.775+0000 7efd03e00700 1 -- 172.21.15.94:0/3580040473 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf8011a20 con 0x7efd0410fc50 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.776+0000 7efd0c000700 1 -- 172.21.15.94:0/3580040473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 msgr2=0x7efd04112040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.776+0000 7efd0c000700 1 --2- 172.21.15.94:0/3580040473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd04112040 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efcf80098a0 tx=0x7efcf8009bb0 comp rx=0 tx=0).stop 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.776+0000 7efd0c000700 1 -- 172.21.15.94:0/3580040473 shutdown_connections 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.776+0000 7efd0c000700 1 --2- 172.21.15.94:0/3580040473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd04112040 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.776+0000 7efd0c000700 1 -- 172.21.15.94:0/3580040473 >> 172.21.15.94:0/3580040473 conn(0x7efd0407e370 msgr2=0x7efd04080790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.776+0000 7efd0c000700 1 -- 172.21.15.94:0/3580040473 shutdown_connections 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.776+0000 7efd0c000700 1 -- 172.21.15.94:0/3580040473 wait complete. 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.777+0000 7efd0c000700 1 Processor -- start 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.777+0000 7efd0c000700 1 -- start start 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.778+0000 7efd0c000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd041a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.778+0000 7efd0c000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efd04113760 con 0x7efd0410fc50 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.778+0000 7efd09600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd041a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:22.844 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.778+0000 7efd09600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd041a31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51564/0 (socket says 172.21.15.94:51564) 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.778+0000 7efd09600700 1 -- 172.21.15.94:0/1358191134 learned_addr learned my addr 172.21.15.94:0/1358191134 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.778+0000 7efd09600700 1 -- 172.21.15.94:0/1358191134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd041a3730 con 0x7efd0410fc50 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.779+0000 7efd09600700 1 --2- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd041a31f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efcf8004660 tx=0x7efcf8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.779+0000 7efd02000700 1 -- 172.21.15.94:0/1358191134 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf8003aa0 con 0x7efd0410fc50 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.779+0000 7efd02000700 1 -- 172.21.15.94:0/1358191134 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcf801b8f0 con 0x7efd0410fc50 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.779+0000 7efd02000700 1 -- 172.21.15.94:0/1358191134 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf8023900 con 0x7efd0410fc50 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.779+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd041a39c0 con 0x7efd0410fc50 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.779+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd041a66b0 con 0x7efd0410fc50 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.780+0000 7efd02000700 1 -- 172.21.15.94:0/1358191134 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efcf8023bc0 con 0x7efd0410fc50 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.780+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd04064a80 con 0x7efd0410fc50 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.781+0000 7efd02000700 1 --2- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcec0396a0 0x7efcec03bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.781+0000 7efd02000700 1 -- 172.21.15.94:0/1358191134 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efcf804e130 con 0x7efd0410fc50 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.781+0000 7efd08c00700 1 --2- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcec0396a0 0x7efcec03bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:22.845 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.782+0000 7efd08c00700 1 --2- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcec0396a0 0x7efcec03bb60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efcf0006fd0 tx=0x7efcf0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:21.786+0000 7efd02000700 1 -- 172.21.15.94:0/1358191134 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efcf801f030 con 0x7efd0410fc50 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.132+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7efd041a6880 con 0x7efd0410fc50 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.136+0000 7efd02000700 1 -- 172.21.15.94:0/1358191134 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7efcf80292a0 con 0x7efd0410fc50 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.138+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcec0396a0 msgr2=0x7efcec03bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.138+0000 7efd0c000700 1 --2- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcec0396a0 0x7efcec03bb60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efcf0006fd0 tx=0x7efcf0006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.138+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 msgr2=0x7efd041a31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.138+0000 7efd0c000700 1 --2- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd041a31f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efcf8004660 tx=0x7efcf8011aa0 comp rx=0 tx=0).stop 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.138+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 shutdown_connections 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.138+0000 7efd0c000700 1 --2- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcec0396a0 0x7efcec03bb60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.138+0000 7efd0c000700 1 --2- 172.21.15.94:0/1358191134 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efd0410fc50 0x7efd041a31f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.138+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 >> 172.21.15.94:0/1358191134 conn(0x7efd0407e370 msgr2=0x7efd0407ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.139+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 shutdown_connections 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr 2024-04-26T20:00:22.139+0000 7efd0c000700 1 -- 172.21.15.94:0/1358191134 wait complete. 2024-04-26T20:00:22.846 INFO:teuthology.orchestra.run.smithi094.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: ceph telemetry on 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout:For more information see: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:22.847 INFO:teuthology.orchestra.run.smithi094.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T20:00:22.848 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:22.848 INFO:teuthology.orchestra.run.smithi094.stdout:Bootstrap complete. 2024-04-26T20:00:22.927 INFO:tasks.cephadm:Fetching config... 2024-04-26T20:00:22.927 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:00:22.927 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T20:00:22.974 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T20:00:22.974 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:00:22.974 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T20:00:23.054 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T20:00:23.054 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:00:23.054 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T20:00:23.171 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T20:00:23.171 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:00:23.171 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T20:00:23.266 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T20:00:23.267 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdGNxZAv1Ik/+3ASg9HX0/PijaT4004dLmquyvCawUgYDK+O92hWCzvd4zY88TFUzKCBypQnqubA+FumYVeBXr76ZikWVgz67jS+EEY56QxVCEqjtXah1NQEEpRgtKkt0I9ZAW8d/ozNQqhXXO7hURnH2jaB4U1QZ1QpiHtDCbUHXU5267b/K5CWVE3b1EOPas/2IxjwVKcAKr1d51208EOcpAsmUpx57DMQiLUuXEkmwybpOMOnflErzglMx/kwKcbAvzDW88IkVE29A4f2PT048sCauM5I/pu/j1hG3CJZPH6D+j5IW+bWx2muHEA2pizMzIepQJtdCBQMO7FH3saHGEZkpVFEPHTMivNjL16/xFHOBz4e/fz4A+lXidWsaCahyFAzJo0iDdbKO4H00UvY8b/dhSz0JOGjjxUYllHabAKvV6H3XrCk2tzoT++hevT21s2o/MktM/RR7ArF3y4gbCDqqBvnOscQmXKMkGZszYcI3vHAxTLoiW7t2N6OM= ceph-420d0bd6-0407-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T20:00:23.376 INFO:teuthology.orchestra.run.smithi094.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdGNxZAv1Ik/+3ASg9HX0/PijaT4004dLmquyvCawUgYDK+O92hWCzvd4zY88TFUzKCBypQnqubA+FumYVeBXr76ZikWVgz67jS+EEY56QxVCEqjtXah1NQEEpRgtKkt0I9ZAW8d/ozNQqhXXO7hURnH2jaB4U1QZ1QpiHtDCbUHXU5267b/K5CWVE3b1EOPas/2IxjwVKcAKr1d51208EOcpAsmUpx57DMQiLUuXEkmwybpOMOnflErzglMx/kwKcbAvzDW88IkVE29A4f2PT048sCauM5I/pu/j1hG3CJZPH6D+j5IW+bWx2muHEA2pizMzIepQJtdCBQMO7FH3saHGEZkpVFEPHTMivNjL16/xFHOBz4e/fz4A+lXidWsaCahyFAzJo0iDdbKO4H00UvY8b/dhSz0JOGjjxUYllHabAKvV6H3XrCk2tzoT++hevT21s2o/MktM/RR7ArF3y4gbCDqqBvnOscQmXKMkGZszYcI3vHAxTLoiW7t2N6OM= ceph-420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:00:23.410 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1358191134' entity='client.admin' 2024-04-26T20:00:23.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdGNxZAv1Ik/+3ASg9HX0/PijaT4004dLmquyvCawUgYDK+O92hWCzvd4zY88TFUzKCBypQnqubA+FumYVeBXr76ZikWVgz67jS+EEY56QxVCEqjtXah1NQEEpRgtKkt0I9ZAW8d/ozNQqhXXO7hURnH2jaB4U1QZ1QpiHtDCbUHXU5267b/K5CWVE3b1EOPas/2IxjwVKcAKr1d51208EOcpAsmUpx57DMQiLUuXEkmwybpOMOnflErzglMx/kwKcbAvzDW88IkVE29A4f2PT048sCauM5I/pu/j1hG3CJZPH6D+j5IW+bWx2muHEA2pizMzIepQJtdCBQMO7FH3saHGEZkpVFEPHTMivNjL16/xFHOBz4e/fz4A+lXidWsaCahyFAzJo0iDdbKO4H00UvY8b/dhSz0JOGjjxUYllHabAKvV6H3XrCk2tzoT++hevT21s2o/MktM/RR7ArF3y4gbCDqqBvnOscQmXKMkGZszYcI3vHAxTLoiW7t2N6OM= ceph-420d0bd6-0407-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T20:00:23.467 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdGNxZAv1Ik/+3ASg9HX0/PijaT4004dLmquyvCawUgYDK+O92hWCzvd4zY88TFUzKCBypQnqubA+FumYVeBXr76ZikWVgz67jS+EEY56QxVCEqjtXah1NQEEpRgtKkt0I9ZAW8d/ozNQqhXXO7hURnH2jaB4U1QZ1QpiHtDCbUHXU5267b/K5CWVE3b1EOPas/2IxjwVKcAKr1d51208EOcpAsmUpx57DMQiLUuXEkmwybpOMOnflErzglMx/kwKcbAvzDW88IkVE29A4f2PT048sCauM5I/pu/j1hG3CJZPH6D+j5IW+bWx2muHEA2pizMzIepQJtdCBQMO7FH3saHGEZkpVFEPHTMivNjL16/xFHOBz4e/fz4A+lXidWsaCahyFAzJo0iDdbKO4H00UvY8b/dhSz0JOGjjxUYllHabAKvV6H3XrCk2tzoT++hevT21s2o/MktM/RR7ArF3y4gbCDqqBvnOscQmXKMkGZszYcI3vHAxTLoiW7t2N6OM= ceph-420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:00:23.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdGNxZAv1Ik/+3ASg9HX0/PijaT4004dLmquyvCawUgYDK+O92hWCzvd4zY88TFUzKCBypQnqubA+FumYVeBXr76ZikWVgz67jS+EEY56QxVCEqjtXah1NQEEpRgtKkt0I9ZAW8d/ozNQqhXXO7hURnH2jaB4U1QZ1QpiHtDCbUHXU5267b/K5CWVE3b1EOPas/2IxjwVKcAKr1d51208EOcpAsmUpx57DMQiLUuXEkmwybpOMOnflErzglMx/kwKcbAvzDW88IkVE29A4f2PT048sCauM5I/pu/j1hG3CJZPH6D+j5IW+bWx2muHEA2pizMzIepQJtdCBQMO7FH3saHGEZkpVFEPHTMivNjL16/xFHOBz4e/fz4A+lXidWsaCahyFAzJo0iDdbKO4H00UvY8b/dhSz0JOGjjxUYllHabAKvV6H3XrCk2tzoT++hevT21s2o/MktM/RR7ArF3y4gbCDqqBvnOscQmXKMkGZszYcI3vHAxTLoiW7t2N6OM= ceph-420d0bd6-0407-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T20:00:23.521 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdGNxZAv1Ik/+3ASg9HX0/PijaT4004dLmquyvCawUgYDK+O92hWCzvd4zY88TFUzKCBypQnqubA+FumYVeBXr76ZikWVgz67jS+EEY56QxVCEqjtXah1NQEEpRgtKkt0I9ZAW8d/ozNQqhXXO7hURnH2jaB4U1QZ1QpiHtDCbUHXU5267b/K5CWVE3b1EOPas/2IxjwVKcAKr1d51208EOcpAsmUpx57DMQiLUuXEkmwybpOMOnflErzglMx/kwKcbAvzDW88IkVE29A4f2PT048sCauM5I/pu/j1hG3CJZPH6D+j5IW+bWx2muHEA2pizMzIepQJtdCBQMO7FH3saHGEZkpVFEPHTMivNjL16/xFHOBz4e/fz4A+lXidWsaCahyFAzJo0iDdbKO4H00UvY8b/dhSz0JOGjjxUYllHabAKvV6H3XrCk2tzoT++hevT21s2o/MktM/RR7ArF3y4gbCDqqBvnOscQmXKMkGZszYcI3vHAxTLoiW7t2N6OM= ceph-420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:00:23.534 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T20:00:23.951 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:00:26.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.734+0000 7fb430800700 1 -- 172.21.15.94:0/1579939863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 msgr2=0x7fb428104900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:26.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.734+0000 7fb430800700 1 --2- 172.21.15.94:0/1579939863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 0x7fb428104900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009910 tx=0x7fb41c009c20 comp rx=0 tx=0).stop 2024-04-26T20:00:26.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.734+0000 7fb430800700 1 -- 172.21.15.94:0/1579939863 shutdown_connections 2024-04-26T20:00:26.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.734+0000 7fb430800700 1 --2- 172.21.15.94:0/1579939863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 0x7fb428104900 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:26.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.734+0000 7fb430800700 1 -- 172.21.15.94:0/1579939863 >> 172.21.15.94:0/1579939863 conn(0x7fb4280fe470 msgr2=0x7fb428100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:26.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.734+0000 7fb430800700 1 -- 172.21.15.94:0/1579939863 shutdown_connections 2024-04-26T20:00:26.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.734+0000 7fb430800700 1 -- 172.21.15.94:0/1579939863 wait complete. 2024-04-26T20:00:26.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.736+0000 7fb430800700 1 Processor -- start 2024-04-26T20:00:26.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.736+0000 7fb430800700 1 -- start start 2024-04-26T20:00:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.736+0000 7fb430800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 0x7fb42819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.736+0000 7fb430800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb428108e50 con 0x7fb4281044e0 2024-04-26T20:00:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.737+0000 7fb42de00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 0x7fb42819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.737+0000 7fb42de00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 0x7fb42819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51584/0 (socket says 172.21.15.94:51584) 2024-04-26T20:00:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.737+0000 7fb42de00700 1 -- 172.21.15.94:0/47967578 learned_addr learned my addr 172.21.15.94:0/47967578 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.737+0000 7fb42de00700 1 -- 172.21.15.94:0/47967578 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42819ad20 con 0x7fb4281044e0 2024-04-26T20:00:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.737+0000 7fb42de00700 1 --2- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 0x7fb42819a7e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb41c005950 tx=0x7fb41c004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.737+0000 7fb422a00700 1 -- 172.21.15.94:0/47967578 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb41c026070 con 0x7fb4281044e0 2024-04-26T20:00:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.738+0000 7fb422a00700 1 -- 172.21.15.94:0/47967578 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb41c00bc00 con 0x7fb4281044e0 2024-04-26T20:00:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.738+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42819afb0 con 0x7fb4281044e0 2024-04-26T20:00:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.738+0000 7fb422a00700 1 -- 172.21.15.94:0/47967578 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb41c003a10 con 0x7fb4281044e0 2024-04-26T20:00:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.738+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb42819dca0 con 0x7fb4281044e0 2024-04-26T20:00:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.739+0000 7fb422a00700 1 -- 172.21.15.94:0/47967578 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb41c029070 con 0x7fb4281044e0 2024-04-26T20:00:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.739+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4281095d0 con 0x7fb4281044e0 2024-04-26T20:00:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.739+0000 7fb422a00700 1 --2- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4100396b0 0x7fb41003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.739+0000 7fb422a00700 1 -- 172.21.15.94:0/47967578 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb41c01e070 con 0x7fb4281044e0 2024-04-26T20:00:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.740+0000 7fb42d400700 1 --2- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4100396b0 0x7fb41003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.740+0000 7fb42d400700 1 --2- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4100396b0 0x7fb41003bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb418006fd0 tx=0x7fb418006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:26.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.745+0000 7fb422a00700 1 -- 172.21.15.94:0/47967578 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb41c013070 con 0x7fb4281044e0 2024-04-26T20:00:26.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.969+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fb42802e490 con 0x7fb4281044e0 2024-04-26T20:00:26.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.975+0000 7fb422a00700 1 -- 172.21.15.94:0/47967578 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fb41c0300c0 con 0x7fb4281044e0 2024-04-26T20:00:26.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.978+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4100396b0 msgr2=0x7fb41003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:26.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.978+0000 7fb430800700 1 --2- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4100396b0 0x7fb41003bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb418006fd0 tx=0x7fb418006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:26.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.978+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 msgr2=0x7fb42819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:26.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.978+0000 7fb430800700 1 --2- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 0x7fb42819a7e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb41c005950 tx=0x7fb41c004fa0 comp rx=0 tx=0).stop 2024-04-26T20:00:26.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.978+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 shutdown_connections 2024-04-26T20:00:26.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.978+0000 7fb430800700 1 --2- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4100396b0 0x7fb41003bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:26.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.978+0000 7fb430800700 1 --2- 172.21.15.94:0/47967578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4281044e0 0x7fb42819a7e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:26.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.978+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 >> 172.21.15.94:0/47967578 conn(0x7fb4280fe470 msgr2=0x7fb4280fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:26.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.979+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 shutdown_connections 2024-04-26T20:00:26.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:26.979+0000 7fb430800700 1 -- 172.21.15.94:0/47967578 wait complete. 2024-04-26T20:00:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi094", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:00:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:27 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/47967578' entity='client.admin' 2024-04-26T20:00:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:00:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:27.725 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T20:00:27.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T20:00:28.150 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:00:30.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.097+0000 7fec43000700 1 -- 172.21.15.94:0/3483470090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 msgr2=0x7fec3c075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:30.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.097+0000 7fec43000700 1 --2- 172.21.15.94:0/3483470090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 0x7fec3c075830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fec30009910 tx=0x7fec30009c20 comp rx=0 tx=0).stop 2024-04-26T20:00:30.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.098+0000 7fec43000700 1 -- 172.21.15.94:0/3483470090 shutdown_connections 2024-04-26T20:00:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.098+0000 7fec43000700 1 --2- 172.21.15.94:0/3483470090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 0x7fec3c075830 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.098+0000 7fec43000700 1 -- 172.21.15.94:0/3483470090 >> 172.21.15.94:0/3483470090 conn(0x7fec3c0fe270 msgr2=0x7fec3c1006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.098+0000 7fec43000700 1 -- 172.21.15.94:0/3483470090 shutdown_connections 2024-04-26T20:00:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.098+0000 7fec43000700 1 -- 172.21.15.94:0/3483470090 wait complete. 2024-04-26T20:00:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.099+0000 7fec43000700 1 Processor -- start 2024-04-26T20:00:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.099+0000 7fec43000700 1 -- start start 2024-04-26T20:00:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.099+0000 7fec43000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 0x7fec3c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.100+0000 7fec43000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c10ae20 con 0x7fec3c0773d0 2024-04-26T20:00:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.100+0000 7fec41c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 0x7fec3c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.100+0000 7fec41c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 0x7fec3c19a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51598/0 (socket says 172.21.15.94:51598) 2024-04-26T20:00:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.100+0000 7fec41c00700 1 -- 172.21.15.94:0/2566420314 learned_addr learned my addr 172.21.15.94:0/2566420314 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.100+0000 7fec41c00700 1 -- 172.21.15.94:0/2566420314 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3c19aa00 con 0x7fec3c0773d0 2024-04-26T20:00:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.101+0000 7fec41c00700 1 --2- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 0x7fec3c19a4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec30005190 tx=0x7fec30004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.101+0000 7fec3a000700 1 -- 172.21.15.94:0/2566420314 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec30027070 con 0x7fec3c0773d0 2024-04-26T20:00:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.101+0000 7fec3a000700 1 -- 172.21.15.94:0/2566420314 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec30022470 con 0x7fec3c0773d0 2024-04-26T20:00:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.101+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec3c19ac90 con 0x7fec3c0773d0 2024-04-26T20:00:30.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.101+0000 7fec3a000700 1 -- 172.21.15.94:0/2566420314 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec3000f5f0 con 0x7fec3c0773d0 2024-04-26T20:00:30.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.101+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec3c19d980 con 0x7fec3c0773d0 2024-04-26T20:00:30.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.102+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec3c10b5a0 con 0x7fec3c0773d0 2024-04-26T20:00:30.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.102+0000 7fec3a000700 1 -- 172.21.15.94:0/2566420314 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fec3002a070 con 0x7fec3c0773d0 2024-04-26T20:00:30.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.103+0000 7fec3a000700 1 --2- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fec24039650 0x7fec2403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:30.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.103+0000 7fec3a000700 1 -- 172.21.15.94:0/2566420314 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fec3001e030 con 0x7fec3c0773d0 2024-04-26T20:00:30.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.103+0000 7fec41200700 1 --2- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fec24039650 0x7fec2403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:30.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.104+0000 7fec41200700 1 --2- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fec24039650 0x7fec2403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fec28006fd0 tx=0x7fec28006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:30.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.109+0000 7fec3a000700 1 -- 172.21.15.94:0/2566420314 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fec30013070 con 0x7fec3c0773d0 2024-04-26T20:00:30.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.337+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fec3c02e5a0 con 0x7fec24039650 2024-04-26T20:00:30.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.342+0000 7fec3a000700 1 -- 172.21.15.94:0/2566420314 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fec3c02e5a0 con 0x7fec24039650 2024-04-26T20:00:30.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.345+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fec24039650 msgr2=0x7fec2403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:30.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.345+0000 7fec43000700 1 --2- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fec24039650 0x7fec2403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fec28006fd0 tx=0x7fec28006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:30.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.345+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 msgr2=0x7fec3c19a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:30.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.345+0000 7fec43000700 1 --2- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 0x7fec3c19a4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec30005190 tx=0x7fec30004f80 comp rx=0 tx=0).stop 2024-04-26T20:00:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.345+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 shutdown_connections 2024-04-26T20:00:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.345+0000 7fec43000700 1 --2- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fec24039650 0x7fec2403bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.345+0000 7fec43000700 1 --2- 172.21.15.94:0/2566420314 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec3c0773d0 0x7fec3c19a4c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.345+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 >> 172.21.15.94:0/2566420314 conn(0x7fec3c0fe270 msgr2=0x7fec3c0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.346+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 shutdown_connections 2024-04-26T20:00:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:30.346+0000 7fec43000700 1 -- 172.21.15.94:0/2566420314 wait complete. 2024-04-26T20:00:31.109 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi103 2024-04-26T20:00:31.109 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T20:00:31.109 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.conf 2024-04-26T20:00:31.127 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T20:00:31.127 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:00:31.185 INFO:tasks.cephadm:Adding host smithi103 to orchestrator... 2024-04-26T20:00:31.186 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch host add smithi103 2024-04-26T20:00:31.601 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:00:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:31 smithi094 ceph-mon[28050]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:00:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:00:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:00:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:00:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:31 smithi094 ceph-mon[28050]: Updating smithi094:/etc/ceph/ceph.conf 2024-04-26T20:00:32.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:32 smithi094 ceph-mon[28050]: Updating smithi094:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:00:32.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:32 smithi094 ceph-mon[28050]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:32.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:32 smithi094 ceph-mon[28050]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:00:33.442 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:33 smithi094 ceph-mon[28050]: Updating smithi094:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T20:00:33.443 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:33 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:33.443 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:33 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:33.443 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:33 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:33.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.611+0000 7fae28a00700 1 -- 172.21.15.94:0/3301867235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 msgr2=0x7fae2410e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:33.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.611+0000 7fae28a00700 1 --2- 172.21.15.94:0/3301867235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 0x7fae2410e7b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fae14009910 tx=0x7fae14009c20 comp rx=0 tx=0).stop 2024-04-26T20:00:33.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.611+0000 7fae28a00700 1 -- 172.21.15.94:0/3301867235 shutdown_connections 2024-04-26T20:00:33.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.611+0000 7fae28a00700 1 --2- 172.21.15.94:0/3301867235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 0x7fae2410e7b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:33.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.611+0000 7fae28a00700 1 -- 172.21.15.94:0/3301867235 >> 172.21.15.94:0/3301867235 conn(0x7fae24078bc0 msgr2=0x7fae2407afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:33.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.612+0000 7fae28a00700 1 -- 172.21.15.94:0/3301867235 shutdown_connections 2024-04-26T20:00:33.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.612+0000 7fae28a00700 1 -- 172.21.15.94:0/3301867235 wait complete. 2024-04-26T20:00:33.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.612+0000 7fae28a00700 1 Processor -- start 2024-04-26T20:00:33.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.613+0000 7fae28a00700 1 -- start start 2024-04-26T20:00:33.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.613+0000 7fae28a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 0x7fae2419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:33.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.613+0000 7fae28a00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fae24113940 con 0x7fae2410e3d0 2024-04-26T20:00:33.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.613+0000 7fae21e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 0x7fae2419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:33.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.613+0000 7fae21e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 0x7fae2419eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48524/0 (socket says 172.21.15.94:48524) 2024-04-26T20:00:33.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.613+0000 7fae21e00700 1 -- 172.21.15.94:0/1317091806 learned_addr learned my addr 172.21.15.94:0/1317091806 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:33.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.614+0000 7fae21e00700 1 -- 172.21.15.94:0/1317091806 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2419f420 con 0x7fae2410e3d0 2024-04-26T20:00:33.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.614+0000 7fae21e00700 1 --2- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 0x7fae2419eee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fae14006010 tx=0x7fae14005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:33.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.614+0000 7fae1aa00700 1 -- 172.21.15.94:0/1317091806 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae14026070 con 0x7fae2410e3d0 2024-04-26T20:00:33.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.614+0000 7fae1aa00700 1 -- 172.21.15.94:0/1317091806 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae1400bba0 con 0x7fae2410e3d0 2024-04-26T20:00:33.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.614+0000 7fae1aa00700 1 -- 172.21.15.94:0/1317091806 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae14003930 con 0x7fae2410e3d0 2024-04-26T20:00:33.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.614+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2419f710 con 0x7fae2410e3d0 2024-04-26T20:00:33.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.615+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae241a37b0 con 0x7fae2410e3d0 2024-04-26T20:00:33.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.616+0000 7fae1aa00700 1 -- 172.21.15.94:0/1317091806 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fae14029070 con 0x7fae2410e3d0 2024-04-26T20:00:33.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.616+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2410ab60 con 0x7fae2410e3d0 2024-04-26T20:00:33.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.616+0000 7fae1aa00700 1 --2- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fae08039700 0x7fae0803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:33.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.616+0000 7fae1aa00700 1 -- 172.21.15.94:0/1317091806 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fae1401e070 con 0x7fae2410e3d0 2024-04-26T20:00:33.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.617+0000 7fae21400700 1 --2- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fae08039700 0x7fae0803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:33.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.617+0000 7fae21400700 1 --2- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fae08039700 0x7fae0803bbc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fae10006fd0 tx=0x7fae10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:33.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.622+0000 7fae1aa00700 1 -- 172.21.15.94:0/1317091806 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fae14013070 con 0x7fae2410e3d0 2024-04-26T20:00:33.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:33.855+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}) -- 0x7fae241a0240 con 0x7fae08039700 2024-04-26T20:00:34.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:34 smithi094 ceph-mon[28050]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:35.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:35 smithi094 ceph-mon[28050]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:00:35.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:35 smithi094 ceph-mon[28050]: Deploying cephadm binary to smithi103 2024-04-26T20:00:36.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:36 smithi094 ceph-mon[28050]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:38.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.539+0000 7fae1aa00700 1 -- 172.21.15.94:0/1317091806 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fae241a0240 con 0x7fae08039700 2024-04-26T20:00:38.541 INFO:teuthology.orchestra.run.smithi094.stdout:Added host 'smithi103' with addr '172.21.15.103' 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.542+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fae08039700 msgr2=0x7fae0803bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.542+0000 7fae28a00700 1 --2- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fae08039700 0x7fae0803bbc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fae10006fd0 tx=0x7fae10006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.542+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 msgr2=0x7fae2419eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.542+0000 7fae28a00700 1 --2- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 0x7fae2419eee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fae14006010 tx=0x7fae14005780 comp rx=0 tx=0).stop 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.542+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 shutdown_connections 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.542+0000 7fae28a00700 1 --2- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fae08039700 0x7fae0803bbc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.542+0000 7fae28a00700 1 --2- 172.21.15.94:0/1317091806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae2410e3d0 0x7fae2419eee0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.542+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 >> 172.21.15.94:0/1317091806 conn(0x7fae24078bc0 msgr2=0x7fae24110f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.543+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 shutdown_connections 2024-04-26T20:00:38.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:38.543+0000 7fae28a00700 1 -- 172.21.15.94:0/1317091806 wait complete. 2024-04-26T20:00:38.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:38 smithi094 ceph-mon[28050]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:39.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T20:00:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:00:39.851 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:39.851 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:39 smithi094 ceph-mon[28050]: Added host smithi103 2024-04-26T20:00:39.851 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:00:39.852 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:40.915 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:40 smithi094 ceph-mon[28050]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:40.915 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:41.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.828+0000 7f910ce00700 1 -- 172.21.15.94:0/916764742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 msgr2=0x7f910810e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:41.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.828+0000 7f910ce00700 1 --2- 172.21.15.94:0/916764742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 0x7f910810e7b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009910 tx=0x7f90f8009c20 comp rx=0 tx=0).stop 2024-04-26T20:00:41.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.829+0000 7f910ce00700 1 -- 172.21.15.94:0/916764742 shutdown_connections 2024-04-26T20:00:41.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.829+0000 7f910ce00700 1 --2- 172.21.15.94:0/916764742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 0x7f910810e7b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:41.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.829+0000 7f910ce00700 1 -- 172.21.15.94:0/916764742 >> 172.21.15.94:0/916764742 conn(0x7f9108078bc0 msgr2=0x7f910807afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:41.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.829+0000 7f910ce00700 1 -- 172.21.15.94:0/916764742 shutdown_connections 2024-04-26T20:00:41.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.829+0000 7f910ce00700 1 -- 172.21.15.94:0/916764742 wait complete. 2024-04-26T20:00:41.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.830+0000 7f910ce00700 1 Processor -- start 2024-04-26T20:00:41.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.830+0000 7f910ce00700 1 -- start start 2024-04-26T20:00:41.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.830+0000 7f910ce00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 0x7f910819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:41.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.830+0000 7f910ce00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9108113940 con 0x7f910810e3d0 2024-04-26T20:00:41.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.831+0000 7f9105e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 0x7f910819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:41.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.831+0000 7f9105e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 0x7f910819ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52384/0 (socket says 172.21.15.94:52384) 2024-04-26T20:00:41.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.831+0000 7f9105e00700 1 -- 172.21.15.94:0/1904142812 learned_addr learned my addr 172.21.15.94:0/1904142812 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:41.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.831+0000 7f9105e00700 1 -- 172.21.15.94:0/1904142812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910819f3b0 con 0x7f910810e3d0 2024-04-26T20:00:41.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.831+0000 7f9105e00700 1 --2- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 0x7f910819ee70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90f8006010 tx=0x7f90f8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:41.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.832+0000 7f90fea00700 1 -- 172.21.15.94:0/1904142812 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90f8026070 con 0x7f910810e3d0 2024-04-26T20:00:41.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.832+0000 7f90fea00700 1 -- 172.21.15.94:0/1904142812 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90f800bba0 con 0x7f910810e3d0 2024-04-26T20:00:41.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.832+0000 7f90fea00700 1 -- 172.21.15.94:0/1904142812 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90f8003930 con 0x7f910810e3d0 2024-04-26T20:00:41.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.832+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910819f6a0 con 0x7f910810e3d0 2024-04-26T20:00:41.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.832+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91081a3740 con 0x7f910810e3d0 2024-04-26T20:00:41.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.833+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910810ab60 con 0x7f910810e3d0 2024-04-26T20:00:41.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.833+0000 7f90fea00700 1 -- 172.21.15.94:0/1904142812 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f90f8029070 con 0x7f910810e3d0 2024-04-26T20:00:41.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.833+0000 7f90fea00700 1 --2- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f90ec0396b0 0x7f90ec03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:41.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.833+0000 7f90fea00700 1 -- 172.21.15.94:0/1904142812 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f90f801e070 con 0x7f910810e3d0 2024-04-26T20:00:41.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.834+0000 7f9105400700 1 --2- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f90ec0396b0 0x7f90ec03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:41.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.834+0000 7f9105400700 1 --2- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f90ec0396b0 0x7f90ec03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f90f4006fd0 tx=0x7f90f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:41.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:41.839+0000 7f90fea00700 1 -- 172.21.15.94:0/1904142812 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f90f8013070 con 0x7f910810e3d0 2024-04-26T20:00:42.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.070+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f91081a01d0 con 0x7f90ec0396b0 2024-04-26T20:00:42.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.073+0000 7f90fea00700 1 -- 172.21.15.94:0/1904142812 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f91081a01d0 con 0x7f90ec0396b0 2024-04-26T20:00:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:42.073 INFO:teuthology.orchestra.run.smithi094.stdout:[{"addr": "172.21.15.94", "hostname": "smithi094", "labels": [], "status": ""}, {"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}] 2024-04-26T20:00:42.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f90ec0396b0 msgr2=0x7f90ec03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:42.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 --2- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f90ec0396b0 0x7f90ec03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f90f4006fd0 tx=0x7f90f4006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:42.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 msgr2=0x7f910819ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:42.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 --2- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 0x7f910819ee70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f90f8006010 tx=0x7f90f8005780 comp rx=0 tx=0).stop 2024-04-26T20:00:42.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 shutdown_connections 2024-04-26T20:00:42.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 --2- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f90ec0396b0 0x7f90ec03bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:42.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 --2- 172.21.15.94:0/1904142812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f910810e3d0 0x7f910819ee70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:42.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 >> 172.21.15.94:0/1904142812 conn(0x7f9108078bc0 msgr2=0x7f9108110f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:42.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 shutdown_connections 2024-04-26T20:00:42.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:42.076+0000 7f910ce00700 1 -- 172.21.15.94:0/1904142812 wait complete. 2024-04-26T20:00:42.402 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:42.402 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:42 smithi094 ceph-mon[28050]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:42.780 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-04-26T20:00:42.780 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T20:00:42.780 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-04-26T20:00:42.799 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T20:00:42.799 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:00:42.856 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-04-26T20:00:42.856 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch host add smithi204 2024-04-26T20:00:43.289 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:00:43.638 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:43 smithi094 ceph-mon[28050]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:00:44.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:44 smithi094 ceph-mon[28050]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:45.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.394+0000 7f187c400700 1 -- 172.21.15.94:0/1387657538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 msgr2=0x7f18741092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:45.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.394+0000 7f187c400700 1 --2- 172.21.15.94:0/1387657538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 0x7f18741092a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1868009910 tx=0x7f1868009c20 comp rx=0 tx=0).stop 2024-04-26T20:00:45.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.394+0000 7f187c400700 1 -- 172.21.15.94:0/1387657538 shutdown_connections 2024-04-26T20:00:45.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.394+0000 7f187c400700 1 --2- 172.21.15.94:0/1387657538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 0x7f18741092a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:45.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.394+0000 7f187c400700 1 -- 172.21.15.94:0/1387657538 >> 172.21.15.94:0/1387657538 conn(0x7f1874077880 msgr2=0x7f1874077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:45.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.394+0000 7f187c400700 1 -- 172.21.15.94:0/1387657538 shutdown_connections 2024-04-26T20:00:45.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.395+0000 7f187c400700 1 -- 172.21.15.94:0/1387657538 wait complete. 2024-04-26T20:00:45.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.396+0000 7f187c400700 1 Processor -- start 2024-04-26T20:00:45.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.396+0000 7f187c400700 1 -- start start 2024-04-26T20:00:45.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.396+0000 7f187c400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 0x7f18741a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:45.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.397+0000 7f187c400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f187410d300 con 0x7f1874108ec0 2024-04-26T20:00:45.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.397+0000 7f1879a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 0x7f18741a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:45.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.397+0000 7f1879a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 0x7f18741a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52414/0 (socket says 172.21.15.94:52414) 2024-04-26T20:00:45.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.397+0000 7f1879a00700 1 -- 172.21.15.94:0/1257665036 learned_addr learned my addr 172.21.15.94:0/1257665036 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:45.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.397+0000 7f1879a00700 1 -- 172.21.15.94:0/1257665036 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18741a0ca0 con 0x7f1874108ec0 2024-04-26T20:00:45.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.398+0000 7f1879a00700 1 --2- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 0x7f18741a0760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1868006010 tx=0x7f1868005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:45.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.398+0000 7f1872000700 1 -- 172.21.15.94:0/1257665036 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1868026070 con 0x7f1874108ec0 2024-04-26T20:00:45.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.398+0000 7f1872000700 1 -- 172.21.15.94:0/1257665036 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186800bba0 con 0x7f1874108ec0 2024-04-26T20:00:45.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.398+0000 7f1872000700 1 -- 172.21.15.94:0/1257665036 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1868003910 con 0x7f1874108ec0 2024-04-26T20:00:45.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.398+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18741a0f90 con 0x7f1874108ec0 2024-04-26T20:00:45.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.398+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f187419a970 con 0x7f1874108ec0 2024-04-26T20:00:45.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.399+0000 7f1872000700 1 -- 172.21.15.94:0/1257665036 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1868029070 con 0x7f1874108ec0 2024-04-26T20:00:45.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.399+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1874104570 con 0x7f1874108ec0 2024-04-26T20:00:45.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.400+0000 7f1872000700 1 --2- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f185c0396b0 0x7f185c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:45.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.400+0000 7f1872000700 1 -- 172.21.15.94:0/1257665036 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f186801e070 con 0x7f1874108ec0 2024-04-26T20:00:45.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.400+0000 7f1879000700 1 --2- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f185c0396b0 0x7f185c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:45.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.401+0000 7f1879000700 1 --2- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f185c0396b0 0x7f185c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1860006fd0 tx=0x7f1860006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:45.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.406+0000 7f1872000700 1 -- 172.21.15.94:0/1257665036 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1868013070 con 0x7f1874108ec0 2024-04-26T20:00:45.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:45.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:45.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:45.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:45.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:45.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:00:45.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:00:45.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:45 smithi094 ceph-mon[28050]: Updating smithi103:/etc/ceph/ceph.conf 2024-04-26T20:00:45.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:45.654+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}) -- 0x7f187419be50 con 0x7f185c0396b0 2024-04-26T20:00:46.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:46 smithi094 ceph-mon[28050]: Updating smithi103:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:00:46.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:46 smithi094 ceph-mon[28050]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:00:46.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:46 smithi094 ceph-mon[28050]: Updating smithi103:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T20:00:46.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:46 smithi094 ceph-mon[28050]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:46.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:46 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:46.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:46 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:46.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:46 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:47.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:47 smithi094 ceph-mon[28050]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:00:47.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:47 smithi094 ceph-mon[28050]: Deploying cephadm binary to smithi204 2024-04-26T20:00:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:48 smithi094 ceph-mon[28050]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.318+0000 7f1872000700 1 -- 172.21.15.94:0/1257665036 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f187419be50 con 0x7f185c0396b0 2024-04-26T20:00:50.320 INFO:teuthology.orchestra.run.smithi094.stdout:Added host 'smithi204' with addr '172.21.15.204' 2024-04-26T20:00:50.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.321+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f185c0396b0 msgr2=0x7f185c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:50.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.321+0000 7f187c400700 1 --2- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f185c0396b0 0x7f185c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1860006fd0 tx=0x7f1860006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:50.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.321+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 msgr2=0x7f18741a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:50.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.321+0000 7f187c400700 1 --2- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 0x7f18741a0760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1868006010 tx=0x7f1868005780 comp rx=0 tx=0).stop 2024-04-26T20:00:50.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.321+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 shutdown_connections 2024-04-26T20:00:50.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.321+0000 7f187c400700 1 --2- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f185c0396b0 0x7f185c03bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:50.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.321+0000 7f187c400700 1 --2- 172.21.15.94:0/1257665036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1874108ec0 0x7f18741a0760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:50.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.321+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 >> 172.21.15.94:0/1257665036 conn(0x7f1874077880 msgr2=0x7f18741052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:50.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.322+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 shutdown_connections 2024-04-26T20:00:50.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:50.322+0000 7f187c400700 1 -- 172.21.15.94:0/1257665036 wait complete. 2024-04-26T20:00:50.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:50 smithi094 ceph-mon[28050]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:50.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:50.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:00:51.119 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T20:00:51.549 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:00:51.850 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:51 smithi094 ceph-mon[28050]: Added host smithi204 2024-04-26T20:00:51.850 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:51 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:51.850 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:51 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:52.841 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:52 smithi094 ceph-mon[28050]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:53.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.613+0000 7ffb72600700 1 -- 172.21.15.94:0/2674384540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 msgr2=0x7ffb6c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:53.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.613+0000 7ffb72600700 1 --2- 172.21.15.94:0/2674384540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 0x7ffb6c10f740 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009910 tx=0x7ffb60009c20 comp rx=0 tx=0).stop 2024-04-26T20:00:53.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.613+0000 7ffb72600700 1 -- 172.21.15.94:0/2674384540 shutdown_connections 2024-04-26T20:00:53.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.613+0000 7ffb72600700 1 --2- 172.21.15.94:0/2674384540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 0x7ffb6c10f740 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:53.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.613+0000 7ffb72600700 1 -- 172.21.15.94:0/2674384540 >> 172.21.15.94:0/2674384540 conn(0x7ffb6c078a30 msgr2=0x7ffb6c07ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:53.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.613+0000 7ffb72600700 1 -- 172.21.15.94:0/2674384540 shutdown_connections 2024-04-26T20:00:53.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.613+0000 7ffb72600700 1 -- 172.21.15.94:0/2674384540 wait complete. 2024-04-26T20:00:53.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.614+0000 7ffb72600700 1 Processor -- start 2024-04-26T20:00:53.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.614+0000 7ffb72600700 1 -- start start 2024-04-26T20:00:53.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.615+0000 7ffb72600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 0x7ffb6c1a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:53.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.615+0000 7ffb72600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c113760 con 0x7ffb6c10f360 2024-04-26T20:00:53.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.615+0000 7ffb71200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 0x7ffb6c1a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:53.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.615+0000 7ffb71200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 0x7ffb6c1a48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41778/0 (socket says 172.21.15.94:41778) 2024-04-26T20:00:53.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.615+0000 7ffb71200700 1 -- 172.21.15.94:0/3095900948 learned_addr learned my addr 172.21.15.94:0/3095900948 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:53.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.615+0000 7ffb71200700 1 -- 172.21.15.94:0/3095900948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6c1a4e30 con 0x7ffb6c10f360 2024-04-26T20:00:53.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.616+0000 7ffb71200700 1 --2- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 0x7ffb6c1a48f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffb60006010 tx=0x7ffb60005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:53.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.616+0000 7ffb69600700 1 -- 172.21.15.94:0/3095900948 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb60026070 con 0x7ffb6c10f360 2024-04-26T20:00:53.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.616+0000 7ffb69600700 1 -- 172.21.15.94:0/3095900948 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb6000bba0 con 0x7ffb6c10f360 2024-04-26T20:00:53.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.616+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6c1a5120 con 0x7ffb6c10f360 2024-04-26T20:00:53.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.616+0000 7ffb69600700 1 -- 172.21.15.94:0/3095900948 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb60003930 con 0x7ffb6c10f360 2024-04-26T20:00:53.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.616+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb6c19eab0 con 0x7ffb6c10f360 2024-04-26T20:00:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.617+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb6c10a9a0 con 0x7ffb6c10f360 2024-04-26T20:00:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.618+0000 7ffb69600700 1 -- 172.21.15.94:0/3095900948 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ffb60029070 con 0x7ffb6c10f360 2024-04-26T20:00:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.618+0000 7ffb69600700 1 --2- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffb540396b0 0x7ffb5403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.618+0000 7ffb69600700 1 -- 172.21.15.94:0/3095900948 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffb6001e070 con 0x7ffb6c10f360 2024-04-26T20:00:53.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.619+0000 7ffb6be00700 1 --2- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffb540396b0 0x7ffb5403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:53.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.619+0000 7ffb6be00700 1 --2- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffb540396b0 0x7ffb5403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ffb58006fd0 tx=0x7ffb58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:53.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.626+0000 7ffb69600700 1 -- 172.21.15.94:0/3095900948 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffb60013070 con 0x7ffb6c10f360 2024-04-26T20:00:53.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.857+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffb6c1a0050 con 0x7ffb540396b0 2024-04-26T20:00:53.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.859+0000 7ffb69600700 1 -- 172.21.15.94:0/3095900948 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7ffb6c1a0050 con 0x7ffb540396b0 2024-04-26T20:00:53.861 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:00:53.861 INFO:teuthology.orchestra.run.smithi094.stdout:[{"addr": "172.21.15.94", "hostname": "smithi094", "labels": [], "status": ""}, {"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.862+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffb540396b0 msgr2=0x7ffb5403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.862+0000 7ffb72600700 1 --2- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffb540396b0 0x7ffb5403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ffb58006fd0 tx=0x7ffb58006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.862+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 msgr2=0x7ffb6c1a48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.862+0000 7ffb72600700 1 --2- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 0x7ffb6c1a48f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffb60006010 tx=0x7ffb60005780 comp rx=0 tx=0).stop 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.862+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 shutdown_connections 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.862+0000 7ffb72600700 1 --2- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffb540396b0 0x7ffb5403bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.862+0000 7ffb72600700 1 --2- 172.21.15.94:0/3095900948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb6c10f360 0x7ffb6c1a48f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.862+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 >> 172.21.15.94:0/3095900948 conn(0x7ffb6c078a30 msgr2=0x7ffb6c07a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.863+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 shutdown_connections 2024-04-26T20:00:53.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:53.863+0000 7ffb72600700 1 -- 172.21.15.94:0/3095900948 wait complete. 2024-04-26T20:00:54.169 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:54.169 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:54 smithi094 ceph-mon[28050]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:54.512 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T20:00:54.512 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T20:00:54.940 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:00:55.315 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:55 smithi094 ceph-mon[28050]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:00:56.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:56 smithi094 ceph-mon[28050]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.114+0000 7fa6ed200700 1 -- 172.21.15.94:0/2660598631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 msgr2=0x7fa6e8075c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.114+0000 7fa6ed200700 1 --2- 172.21.15.94:0/2660598631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 0x7fa6e8075c30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009910 tx=0x7fa6d8009c20 comp rx=0 tx=0).stop 2024-04-26T20:00:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.114+0000 7fa6ed200700 1 -- 172.21.15.94:0/2660598631 shutdown_connections 2024-04-26T20:00:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.114+0000 7fa6ed200700 1 --2- 172.21.15.94:0/2660598631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 0x7fa6e8075c30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.114+0000 7fa6ed200700 1 -- 172.21.15.94:0/2660598631 >> 172.21.15.94:0/2660598631 conn(0x7fa6e80fe960 msgr2=0x7fa6e8100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.115+0000 7fa6ed200700 1 -- 172.21.15.94:0/2660598631 shutdown_connections 2024-04-26T20:00:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.115+0000 7fa6ed200700 1 -- 172.21.15.94:0/2660598631 wait complete. 2024-04-26T20:00:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.116+0000 7fa6ed200700 1 Processor -- start 2024-04-26T20:00:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.117+0000 7fa6ed200700 1 -- start start 2024-04-26T20:00:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.117+0000 7fa6ed200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 0x7fa6e81a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.117+0000 7fa6ed200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e810f590 con 0x7fa6e8075850 2024-04-26T20:00:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.117+0000 7fa6e5e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 0x7fa6e81a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.117+0000 7fa6e5e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 0x7fa6e81a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41794/0 (socket says 172.21.15.94:41794) 2024-04-26T20:00:57.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.117+0000 7fa6e5e00700 1 -- 172.21.15.94:0/934015672 learned_addr learned my addr 172.21.15.94:0/934015672 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:00:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.118+0000 7fa6e5e00700 1 -- 172.21.15.94:0/934015672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e81a0c90 con 0x7fa6e8075850 2024-04-26T20:00:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.118+0000 7fa6e5e00700 1 --2- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 0x7fa6e81a0750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8006010 tx=0x7fa6d8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.118+0000 7fa6dea00700 1 -- 172.21.15.94:0/934015672 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6d8026070 con 0x7fa6e8075850 2024-04-26T20:00:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.119+0000 7fa6dea00700 1 -- 172.21.15.94:0/934015672 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6d800bc00 con 0x7fa6e8075850 2024-04-26T20:00:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.119+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e81a0f20 con 0x7fa6e8075850 2024-04-26T20:00:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.119+0000 7fa6dea00700 1 -- 172.21.15.94:0/934015672 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6d8003910 con 0x7fa6e8075850 2024-04-26T20:00:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.119+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e819a910 con 0x7fa6e8075850 2024-04-26T20:00:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.120+0000 7fa6dea00700 1 -- 172.21.15.94:0/934015672 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa6d8029070 con 0x7fa6e8075850 2024-04-26T20:00:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.120+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e8076f00 con 0x7fa6e8075850 2024-04-26T20:00:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.120+0000 7fa6dea00700 1 --2- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa6cc039650 0x7fa6cc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:00:57.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.120+0000 7fa6dea00700 1 -- 172.21.15.94:0/934015672 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa6d801e070 con 0x7fa6e8075850 2024-04-26T20:00:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.121+0000 7fa6dfe00700 1 --2- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa6cc039650 0x7fa6cc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:00:57.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.121+0000 7fa6dfe00700 1 --2- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa6cc039650 0x7fa6cc03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0006fd0 tx=0x7fa6d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:00:57.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.126+0000 7fa6dea00700 1 -- 172.21.15.94:0/934015672 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa6d8013070 con 0x7fa6e8075850 2024-04-26T20:00:57.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:57.360+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa6e8064a80 con 0x7fa6e8075850 2024-04-26T20:00:57.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:57.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:57.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:57.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:57.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:00:57.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:00:57.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:00:57.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:57 smithi094 ceph-mon[28050]: Updating smithi204:/etc/ceph/ceph.conf 2024-04-26T20:00:57.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:57 smithi094 ceph-mon[28050]: Updating smithi204:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:00:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.303+0000 7fa6dea00700 1 -- 172.21.15.94:0/934015672 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fa6d800fe20 con 0x7fa6e8075850 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa6cc039650 msgr2=0x7fa6cc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 --2- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa6cc039650 0x7fa6cc03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0006fd0 tx=0x7fa6d0006e40 comp rx=0 tx=0).stop 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 msgr2=0x7fa6e81a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 --2- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 0x7fa6e81a0750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8006010 tx=0x7fa6d8005780 comp rx=0 tx=0).stop 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 shutdown_connections 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 --2- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa6cc039650 0x7fa6cc03bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 --2- 172.21.15.94:0/934015672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6e8075850 0x7fa6e81a0750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 >> 172.21.15.94:0/934015672 conn(0x7fa6e80fe960 msgr2=0x7fa6e81070d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 shutdown_connections 2024-04-26T20:00:58.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:00:58.307+0000 7fa6ed200700 1 -- 172.21.15.94:0/934015672 wait complete. 2024-04-26T20:00:58.310 INFO:teuthology.orchestra.run.smithi094.stderr:adjusted tunables profile to default 2024-04-26T20:00:58.674 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:58 smithi094 ceph-mon[28050]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:00:58.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:58 smithi094 ceph-mon[28050]: Updating smithi204:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T20:00:58.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:58.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:58.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:00:58.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:58 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/934015672' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T20:00:58.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:58 smithi094 ceph-mon[28050]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:00:59.023 INFO:tasks.cephadm:Adding mon.a on smithi094 2024-04-26T20:00:59.023 INFO:tasks.cephadm:Adding mon.b on smithi103 2024-04-26T20:00:59.023 INFO:tasks.cephadm:Adding mon.c on smithi204 2024-04-26T20:00:59.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi094:172.21.15.94=a;smithi103:172.21.15.103=b;smithi204:172.21.15.204=c' 2024-04-26T20:00:59.260 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:00:59.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:59 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/934015672' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T20:00:59.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:00:59 smithi094 ceph-mon[28050]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:01:00.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.701+0000 7f844b09e700 1 -- 172.21.15.204:0/124473603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 msgr2=0x7f8444102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:00.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.701+0000 7f844b09e700 1 --2- 172.21.15.204:0/124473603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 0x7f8444102370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8438009910 tx=0x7f8438009c20 comp rx=0 tx=0).stop 2024-04-26T20:01:00.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.701+0000 7f844b09e700 1 -- 172.21.15.204:0/124473603 shutdown_connections 2024-04-26T20:01:00.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.701+0000 7f844b09e700 1 --2- 172.21.15.204:0/124473603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 0x7f8444102370 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:00.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.701+0000 7f844b09e700 1 -- 172.21.15.204:0/124473603 >> 172.21.15.204:0/124473603 conn(0x7f84440fd840 msgr2=0x7f84440ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:00.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.701+0000 7f844b09e700 1 -- 172.21.15.204:0/124473603 shutdown_connections 2024-04-26T20:01:00.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.701+0000 7f844b09e700 1 -- 172.21.15.204:0/124473603 wait complete. 2024-04-26T20:01:00.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.702+0000 7f844b09e700 1 Processor -- start 2024-04-26T20:01:00.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.702+0000 7f844b09e700 1 -- start start 2024-04-26T20:01:00.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.702+0000 7f844b09e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 0x7f844419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:00.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.702+0000 7f844b09e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f844410c210 con 0x7f8444101f90 2024-04-26T20:01:00.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.702+0000 7f8448e3a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 0x7f844419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:00.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.703+0000 7f8448e3a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 0x7f844419d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.204:47642/0 (socket says 172.21.15.204:47642) 2024-04-26T20:01:00.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.703+0000 7f8448e3a700 1 -- 172.21.15.204:0/737743168 learned_addr learned my addr 172.21.15.204:0/737743168 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:01:00.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.703+0000 7f8448e3a700 1 -- 172.21.15.204:0/737743168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844419d910 con 0x7f8444101f90 2024-04-26T20:01:00.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.704+0000 7f8448e3a700 1 --2- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 0x7f844419d3d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f84380056d0 tx=0x7f84380057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:00.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.704+0000 7f8441ffb700 1 -- 172.21.15.204:0/737743168 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8438026070 con 0x7f8444101f90 2024-04-26T20:01:00.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.704+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f844419dc00 con 0x7f8444101f90 2024-04-26T20:01:00.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.704+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8444197590 con 0x7f8444101f90 2024-04-26T20:01:00.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.704+0000 7f8441ffb700 1 -- 172.21.15.204:0/737743168 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f843800b6f0 con 0x7f8444101f90 2024-04-26T20:01:00.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.704+0000 7f8441ffb700 1 -- 172.21.15.204:0/737743168 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8438030060 con 0x7f8444101f90 2024-04-26T20:01:00.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.705+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8428005330 con 0x7f8444101f90 2024-04-26T20:01:00.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.706+0000 7f8441ffb700 1 -- 172.21.15.204:0/737743168 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8438029070 con 0x7f8444101f90 2024-04-26T20:01:00.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.706+0000 7f8441ffb700 1 --2- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8434039630 0x7f843403baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:00.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.706+0000 7f8441ffb700 1 -- 172.21.15.204:0/737743168 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f843801e070 con 0x7f8444101f90 2024-04-26T20:01:00.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.706+0000 7f8443fff700 1 --2- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8434039630 0x7f843403baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:00.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.707+0000 7f8443fff700 1 --2- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8434039630 0x7f843403baf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8430006fd0 tx=0x7f8430006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:00.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.722+0000 7f8441ffb700 1 -- 172.21.15.204:0/737743168 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8438013070 con 0x7f8444101f90 2024-04-26T20:01:00.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:00 smithi094 ceph-mon[28050]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:00.932 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.931+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi094:172.21.15.94=a;smithi103:172.21.15.103=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}) -- 0x7f8428000c80 con 0x7f8434039630 2024-04-26T20:01:00.947 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.946+0000 7f8441ffb700 1 -- 172.21.15.204:0/737743168 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8428000c80 con 0x7f8434039630 2024-04-26T20:01:00.947 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mon update... 2024-04-26T20:01:00.949 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8434039630 msgr2=0x7f843403baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:00.949 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 --2- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8434039630 0x7f843403baf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8430006fd0 tx=0x7f8430006e40 comp rx=0 tx=0).stop 2024-04-26T20:01:00.949 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 msgr2=0x7f844419d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:00.949 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 --2- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 0x7f844419d3d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f84380056d0 tx=0x7f84380057b0 comp rx=0 tx=0).stop 2024-04-26T20:01:00.950 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 shutdown_connections 2024-04-26T20:01:00.950 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 --2- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8434039630 0x7f843403baf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:00.950 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 --2- 172.21.15.204:0/737743168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8444101f90 0x7f844419d3d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:00.950 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 >> 172.21.15.204:0/737743168 conn(0x7f84440fd840 msgr2=0x7f8444103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:00.950 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 shutdown_connections 2024-04-26T20:01:00.950 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:00.948+0000 7f844b09e700 1 -- 172.21.15.204:0/737743168 wait complete. 2024-04-26T20:01:01.547 DEBUG:teuthology.orchestra.run.smithi103:mon.b> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T20:01:01.550 DEBUG:teuthology.orchestra.run.smithi204:mon.c> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T20:01:01.552 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T20:01:01.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T20:01:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi094:172.21.15.94=a;smithi103:172.21.15.103=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: Saving service mon spec with placement smithi094:172.21.15.94=a;smithi103:172.21.15.103=b;smithi204:172.21.15.204=c;count:3 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: Deploying daemon mon.c on smithi204 2024-04-26T20:01:02.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:01 smithi094 ceph-mon[28050]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:02.739 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:04.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:04 smithi094 ceph-mon[28050]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:05.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.443+0000 7f5a7816e700 1 -- 172.21.15.204:0/3621626700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 msgr2=0x7f5a70109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:05.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.443+0000 7f5a7816e700 1 --2- 172.21.15.204:0/3621626700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 0x7f5a70109390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009910 tx=0x7f5a60009c20 comp rx=0 tx=0).stop 2024-04-26T20:01:05.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.443+0000 7f5a7816e700 1 -- 172.21.15.204:0/3621626700 shutdown_connections 2024-04-26T20:01:05.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.443+0000 7f5a7816e700 1 --2- 172.21.15.204:0/3621626700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 0x7f5a70109390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:05.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.443+0000 7f5a7816e700 1 -- 172.21.15.204:0/3621626700 >> 172.21.15.204:0/3621626700 conn(0x7f5a700784a0 msgr2=0x7f5a700788b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:05.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.443+0000 7f5a7816e700 1 -- 172.21.15.204:0/3621626700 shutdown_connections 2024-04-26T20:01:05.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.443+0000 7f5a7816e700 1 -- 172.21.15.204:0/3621626700 wait complete. 2024-04-26T20:01:05.446 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.443+0000 7f5a7816e700 1 Processor -- start 2024-04-26T20:01:05.446 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.443+0000 7f5a7816e700 1 -- start start 2024-04-26T20:01:05.446 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.444+0000 7f5a7816e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 0x7f5a7019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:05.446 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.444+0000 7f5a7816e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010d460 con 0x7f5a70108fb0 2024-04-26T20:01:05.446 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.444+0000 7f5a75f0a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 0x7f5a7019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:05.446 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.444+0000 7f5a75f0a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 0x7f5a7019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.204:41610/0 (socket says 172.21.15.204:41610) 2024-04-26T20:01:05.446 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.444+0000 7f5a75f0a700 1 -- 172.21.15.204:0/800743304 learned_addr learned my addr 172.21.15.204:0/800743304 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:01:05.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.445+0000 7f5a75f0a700 1 -- 172.21.15.204:0/800743304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7019ebc0 con 0x7f5a70108fb0 2024-04-26T20:01:05.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.445+0000 7f5a75f0a700 1 --2- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 0x7f5a7019e680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5a600056d0 tx=0x7f5a600057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:05.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.445+0000 7f5a66ffd700 1 -- 172.21.15.204:0/800743304 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a60026070 con 0x7f5a70108fb0 2024-04-26T20:01:05.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.446+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7019ee50 con 0x7f5a70108fb0 2024-04-26T20:01:05.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.446+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a70198750 con 0x7f5a70108fb0 2024-04-26T20:01:05.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.446+0000 7f5a66ffd700 1 -- 172.21.15.204:0/800743304 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6000b750 con 0x7f5a70108fb0 2024-04-26T20:01:05.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.446+0000 7f5a66ffd700 1 -- 172.21.15.204:0/800743304 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a60030060 con 0x7f5a70108fb0 2024-04-26T20:01:05.448 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.446+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a54005330 con 0x7f5a70108fb0 2024-04-26T20:01:05.448 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.447+0000 7f5a66ffd700 1 -- 172.21.15.204:0/800743304 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5a60029070 con 0x7f5a70108fb0 2024-04-26T20:01:05.448 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.447+0000 7f5a66ffd700 1 --2- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5a5c039650 0x7f5a5c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:05.448 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.447+0000 7f5a66ffd700 1 -- 172.21.15.204:0/800743304 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5a6001e070 con 0x7f5a70108fb0 2024-04-26T20:01:05.449 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.448+0000 7f5a75709700 1 --2- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5a5c039650 0x7f5a5c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:05.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.449+0000 7f5a75709700 1 --2- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5a5c039650 0x7f5a5c03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c006fd0 tx=0x7f5a6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:05.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.463+0000 7f5a66ffd700 1 -- 172.21.15.204:0/800743304 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5a60013070 con 0x7f5a70108fb0 2024-04-26T20:01:05.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.782+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5a540058f0 con 0x7f5a70108fb0 2024-04-26T20:01:05.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.783+0000 7f5a66ffd700 1 -- 172.21.15.204:0/800743304 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f5a6000e840 con 0x7f5a70108fb0 2024-04-26T20:01:05.784 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T20:01:05.784 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":1,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","modified":"2024-04-26T19:58:53.653581Z","created":"2024-04-26T19:58:53.653581Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:3300","nonce":0},{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T20:01:05.786 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5a5c039650 msgr2=0x7f5a5c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:05.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 --2- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5a5c039650 0x7f5a5c03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c006fd0 tx=0x7f5a6c006e40 comp rx=0 tx=0).stop 2024-04-26T20:01:05.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 msgr2=0x7f5a7019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:05.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 --2- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 0x7f5a7019e680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5a600056d0 tx=0x7f5a600057b0 comp rx=0 tx=0).stop 2024-04-26T20:01:05.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 shutdown_connections 2024-04-26T20:01:05.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 --2- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5a5c039650 0x7f5a5c03bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:05.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 --2- 172.21.15.204:0/800743304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a70108fb0 0x7f5a7019e680 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:05.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 >> 172.21.15.204:0/800743304 conn(0x7f5a700784a0 msgr2=0x7f5a70104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:05.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 shutdown_connections 2024-04-26T20:01:05.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:05.785+0000 7f5a7816e700 1 -- 172.21.15.204:0/800743304 wait complete. 2024-04-26T20:01:05.788 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 1 2024-04-26T20:01:07.509 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T20:01:07.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T20:01:07.755 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:01:11.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:11.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:01:11.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: mon.a calling monitor election 2024-04-26T20:01:11.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:11.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:11.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:11.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:11.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: mon.c calling monitor election 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: monmap epoch 2 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: last_changed 2024-04-26T20:01:05.920088+0000 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: created 2024-04-26T19:58:53.653581+0000 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: min_mon_release 19 (squid) 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: election_strategy: 1 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: fsmap 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:01:11.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: mgrmap e13: x(active, since 59s) 2024-04-26T20:01:11.490 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: overall HEALTH_OK 2024-04-26T20:01:11.490 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:11.490 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:11.490 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:11.490 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:12.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.108+0000 7f14a7f28700 1 -- 172.21.15.204:0/1871363517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f148c0035c0 msgr2=0x7f148c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:12.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.108+0000 7f14a7f28700 1 --2- 172.21.15.204:0/1871363517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f148c0035c0 0x7f148c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f14a007a660 tx=0x7f1498009e80 comp rx=0 tx=0).stop 2024-04-26T20:01:12.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.108+0000 7f14a7f28700 1 -- 172.21.15.204:0/1871363517 shutdown_connections 2024-04-26T20:01:12.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.108+0000 7f14a7f28700 1 --2- 172.21.15.204:0/1871363517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f148c0035c0 0x7f148c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:12.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.108+0000 7f14a7f28700 1 -- 172.21.15.204:0/1871363517 >> 172.21.15.204:0/1871363517 conn(0x7f14a00784d0 msgr2=0x7f14a00788e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:12.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.109+0000 7f14a7f28700 1 -- 172.21.15.204:0/1871363517 shutdown_connections 2024-04-26T20:01:12.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.109+0000 7f14a7f28700 1 -- 172.21.15.204:0/1871363517 wait complete. 2024-04-26T20:01:12.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.109+0000 7f14a7f28700 1 Processor -- start 2024-04-26T20:01:12.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a7f28700 1 -- start start 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a7f28700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 0x7f14a01a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a7f28700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a01a0e40 0x7f14a019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a7f28700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f14a0109e20 con 0x7f14a01a0e40 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a7f28700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f14a0109ca0 con 0x7f148c0035c0 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a5cc4700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 0x7f14a01a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a5cc4700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 0x7f14a01a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:55512/0 (socket says 172.21.15.204:55512) 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a5cc4700 1 -- 172.21.15.204:0/1445650789 learned_addr learned my addr 172.21.15.204:0/1445650789 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a5cc4700 1 -- 172.21.15.204:0/1445650789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 msgr2=0x7f14a01a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a5cc4700 1 -- 172.21.15.204:0/1445650789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 msgr2=0x7f14a01a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a5cc4700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 0x7f14a01a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a5cc4700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 0x7f14a01a0900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:01:12.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.110+0000 7f14a54c3700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a01a0e40 0x7f14a019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:12.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.111+0000 7f14a54c3700 1 -- 172.21.15.204:0/1445650789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 msgr2=0x7f14a01a0900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:01:12.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.111+0000 7f14a54c3700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 0x7f14a01a0900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:12.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.111+0000 7f14a54c3700 1 -- 172.21.15.204:0/1445650789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a01a1480 con 0x7f14a01a0e40 2024-04-26T20:01:12.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.111+0000 7f14a54c3700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a01a0e40 0x7f14a019ab10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f149000b4d0 tx=0x7f149000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:12.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.112+0000 7f1496ffd700 1 -- 172.21.15.204:0/1445650789 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f1490011630 con 0x7f14a01a0e40 2024-04-26T20:01:12.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.112+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14a019b050 con 0x7f14a01a0e40 2024-04-26T20:01:12.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.112+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14a019b5e0 con 0x7f14a01a0e40 2024-04-26T20:01:12.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.112+0000 7f1496ffd700 1 -- 172.21.15.204:0/1445650789 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1490011c80 con 0x7f14a01a0e40 2024-04-26T20:01:12.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.112+0000 7f1496ffd700 1 -- 172.21.15.204:0/1445650789 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f149001ad00 con 0x7f14a01a0e40 2024-04-26T20:01:12.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.113+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f148c002160 con 0x7f14a01a0e40 2024-04-26T20:01:12.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.113+0000 7f1496ffd700 1 -- 172.21.15.204:0/1445650789 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1490018020 con 0x7f14a01a0e40 2024-04-26T20:01:12.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.114+0000 7f1496ffd700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1484039460 0x7f148403b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:12.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.114+0000 7f1496ffd700 1 -- 172.21.15.204:0/1445650789 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f149004c4f0 con 0x7f14a01a0e40 2024-04-26T20:01:12.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.114+0000 7f14a5cc4700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1484039460 0x7f148403b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:12.116 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.115+0000 7f14a5cc4700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1484039460 0x7f148403b920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1498007590 tx=0x7f14980074e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:12.129 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.128+0000 7f1496ffd700 1 -- 172.21.15.204:0/1445650789 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f149001f020 con 0x7f14a01a0e40 2024-04-26T20:01:12.412 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.411+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f148c002c50 con 0x7f14a01a0e40 2024-04-26T20:01:12.413 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.412+0000 7f1496ffd700 1 -- 172.21.15.204:0/1445650789 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f1490014170 con 0x7f14a01a0e40 2024-04-26T20:01:12.414 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T20:01:12.414 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","modified":"2024-04-26T20:01:05.920088Z","created":"2024-04-26T19:58:53.653581Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:3300","nonce":0},{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1484039460 msgr2=0x7f148403b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1484039460 0x7f148403b920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1498007590 tx=0x7f14980074e0 comp rx=0 tx=0).stop 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a01a0e40 msgr2=0x7f14a019ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a01a0e40 0x7f14a019ab10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f149000b4d0 tx=0x7f149000b890 comp rx=0 tx=0).stop 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 shutdown_connections 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1484039460 0x7f148403b920 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f148c0035c0 0x7f14a01a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 --2- 172.21.15.204:0/1445650789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a01a0e40 0x7f14a019ab10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 >> 172.21.15.204:0/1445650789 conn(0x7f14a00784d0 msgr2=0x7f14a01015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:12.416 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 shutdown_connections 2024-04-26T20:01:12.417 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:12.415+0000 7f14a7f28700 1 -- 172.21.15.204:0/1445650789 wait complete. 2024-04-26T20:01:12.417 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-04-26T20:01:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:12 smithi094 ceph-mon[28050]: Deploying daemon mon.b on smithi103 2024-04-26T20:01:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:12 smithi094 ceph-mon[28050]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:13.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:13 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/1445650789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T20:01:14.032 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T20:01:14.032 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T20:01:14.276 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:01:14.389 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:14 smithi094 ceph-mon[28050]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:15.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:14 smithi103 systemd[1]: Started Ceph mon.b for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:01:20.012 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.011+0000 7f83a3835700 1 -- 172.21.15.204:0/2640536727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83880035c0 msgr2=0x7f8388005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:20.012 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.011+0000 7f83a3835700 1 --2- 172.21.15.204:0/2640536727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83880035c0 0x7f8388005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f839c102e70 tx=0x7f839800fe60 comp rx=0 tx=0).stop 2024-04-26T20:01:20.013 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.011+0000 7f83a3835700 1 -- 172.21.15.204:0/2640536727 shutdown_connections 2024-04-26T20:01:20.013 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.011+0000 7f83a3835700 1 --2- 172.21.15.204:0/2640536727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83880035c0 0x7f8388005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:20.013 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.011+0000 7f83a3835700 1 -- 172.21.15.204:0/2640536727 >> 172.21.15.204:0/2640536727 conn(0x7f839c0fea30 msgr2=0x7f839c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:20.013 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.011+0000 7f83a3835700 1 -- 172.21.15.204:0/2640536727 shutdown_connections 2024-04-26T20:01:20.013 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.011+0000 7f83a3835700 1 -- 172.21.15.204:0/2640536727 wait complete. 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.012+0000 7f83a3835700 1 Processor -- start 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.012+0000 7f83a3835700 1 -- start start 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.012+0000 7f83a3835700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83880035c0 0x7f839c075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.012+0000 7f83a3835700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839c075c60 0x7f839c0760e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.012+0000 7f83a3835700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 0x7f839c1a70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.012+0000 7f83a3835700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10ae30 con 0x7f83880035c0 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.012+0000 7f83a3835700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10acb0 con 0x7f839c075c60 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.012+0000 7f83a3835700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10afb0 con 0x7f839c1a4cf0 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a1dd2700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 0x7f839c1a70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a1dd2700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 0x7f839c1a70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:34978/0 (socket says 172.21.15.204:34978) 2024-04-26T20:01:20.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a1dd2700 1 -- 172.21.15.204:0/1849934438 learned_addr learned my addr 172.21.15.204:0/1849934438 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a0dd0700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839c075c60 0x7f839c0760e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a1dd2700 1 -- 172.21.15.204:0/1849934438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 msgr2=0x7f839c1a70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a1dd2700 1 -- 172.21.15.204:0/1849934438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 msgr2=0x7f839c1a70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a1dd2700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 0x7f839c1a70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a1dd2700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 0x7f839c1a70e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a15d1700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83880035c0 0x7f839c075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a0dd0700 1 -- 172.21.15.204:0/1849934438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839c075c60 msgr2=0x7f839c0760e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a0dd0700 1 -- 172.21.15.204:0/1849934438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839c075c60 msgr2=0x7f839c0760e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a0dd0700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839c075c60 0x7f839c0760e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a0dd0700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839c075c60 0x7f839c0760e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a15d1700 1 -- 172.21.15.204:0/1849934438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 msgr2=0x7f839c1a70e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a15d1700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 0x7f839c1a70e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a15d1700 1 -- 172.21.15.204:0/1849934438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839c075c60 msgr2=0x7f839c0760e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:01:20.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.013+0000 7f83a15d1700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839c075c60 0x7f839c0760e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:20.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.014+0000 7f83a15d1700 1 -- 172.21.15.204:0/1849934438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c1a77b0 con 0x7f83880035c0 2024-04-26T20:01:20.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.014+0000 7f83a15d1700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83880035c0 0x7f839c075720 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f839801f000 tx=0x7f8398003ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:20.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.014+0000 7f83927fc700 1 -- 172.21.15.204:0/1849934438 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839802b020 con 0x7f83880035c0 2024-04-26T20:01:20.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.014+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839c1a7a40 con 0x7f83880035c0 2024-04-26T20:01:20.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.014+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839c1a7fd0 con 0x7f83880035c0 2024-04-26T20:01:20.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.014+0000 7f83927fc700 1 -- 172.21.15.204:0/1849934438 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83980056c0 con 0x7f83880035c0 2024-04-26T20:01:20.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.014+0000 7f83927fc700 1 -- 172.21.15.204:0/1849934438 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398022040 con 0x7f83880035c0 2024-04-26T20:01:20.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.016+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839c064a80 con 0x7f83880035c0 2024-04-26T20:01:20.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.016+0000 7f83927fc700 1 -- 172.21.15.204:0/1849934438 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f839802e020 con 0x7f83880035c0 2024-04-26T20:01:20.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.016+0000 7f83927fc700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8380039ae0 0x7f838003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:20.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.016+0000 7f83927fc700 1 -- 172.21.15.204:0/1849934438 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f839804f240 con 0x7f83880035c0 2024-04-26T20:01:20.019 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.017+0000 7f83a0dd0700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8380039ae0 0x7f838003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:20.019 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.018+0000 7f83a0dd0700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8380039ae0 0x7f838003bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f838c009850 tx=0x7f838c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:20.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.032+0000 7f83927fc700 1 -- 172.21.15.204:0/1849934438 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8398035020 con 0x7f83880035c0 2024-04-26T20:01:20.319 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.318+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f839c1a82a0 con 0x7f83880035c0 2024-04-26T20:01:20.320 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.319+0000 7f83927fc700 1 -- 172.21.15.204:0/1849934438 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f8398038af0 con 0x7f83880035c0 2024-04-26T20:01:20.320 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-26T20:01:20.321 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","modified":"2024-04-26T20:01:14.970806Z","created":"2024-04-26T19:58:53.653581Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:3300","nonce":0},{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T20:01:20.323 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8380039ae0 msgr2=0x7f838003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:20.323 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8380039ae0 0x7f838003bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f838c009850 tx=0x7f838c008040 comp rx=0 tx=0).stop 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83880035c0 msgr2=0x7f839c075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83880035c0 0x7f839c075720 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f839801f000 tx=0x7f8398003ca0 comp rx=0 tx=0).stop 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 shutdown_connections 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c1a4cf0 0x7f839c1a70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83880035c0 0x7f839c075720 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8380039ae0 0x7f838003bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 --2- 172.21.15.204:0/1849934438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839c075c60 0x7f839c0760e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 >> 172.21.15.204:0/1849934438 conn(0x7f839c0fea30 msgr2=0x7f839c106b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 shutdown_connections 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:20.322+0000 7f83a3835700 1 -- 172.21.15.204:0/1849934438 wait complete. 2024-04-26T20:01:20.324 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-04-26T20:01:20.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: mon.c calling monitor election 2024-04-26T20:01:20.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: mon.a calling monitor election 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: monmap epoch 3 2024-04-26T20:01:20.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: last_changed 2024-04-26T20:01:14.970806+0000 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: created 2024-04-26T19:58:53.653581+0000 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: min_mon_release 19 (squid) 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: election_strategy: 1 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: fsmap 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: mgrmap e13: x(active, since 68s) 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: overall HEALTH_OK 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:20.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:20.864 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T20:01:20.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T20:01:21.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:21 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/1849934438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T20:01:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:21.297 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: mon.c calling monitor election 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: mon.a calling monitor election 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:22.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: monmap epoch 3 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: last_changed 2024-04-26T20:01:14.970806+0000 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: created 2024-04-26T19:58:53.653581+0000 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: min_mon_release 19 (squid) 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: election_strategy: 1 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-26T20:01:22.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: fsmap 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: mgrmap e13: x(active, since 68s) 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: overall HEALTH_OK 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='client.? 172.21.15.204:0/1849934438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:22.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:22.824 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: mon.b calling monitor election 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: mon.b calling monitor election 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: mon.c calling monitor election 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: mon.a calling monitor election 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: monmap epoch 3 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: last_changed 2024-04-26T20:01:14.970806+0000 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: created 2024-04-26T19:58:53.653581+0000 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: min_mon_release 19 (squid) 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: election_strategy: 1 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: fsmap 2024-04-26T20:01:22.825 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:01:22.826 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: mgrmap e13: x(active, since 70s) 2024-04-26T20:01:22.826 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: overall HEALTH_OK 2024-04-26T20:01:22.826 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.826 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.826 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.826 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: mon.b calling monitor election 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: mon.b calling monitor election 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: mon.c calling monitor election 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: mon.a calling monitor election 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: monmap epoch 3 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: last_changed 2024-04-26T20:01:14.970806+0000 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: created 2024-04-26T19:58:53.653581+0000 2024-04-26T20:01:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: min_mon_release 19 (squid) 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: election_strategy: 1 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: fsmap 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: mgrmap e13: x(active, since 70s) 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: overall HEALTH_OK 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:22.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: mon.b calling monitor election 2024-04-26T20:01:23.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: mon.b calling monitor election 2024-04-26T20:01:23.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: mon.c calling monitor election 2024-04-26T20:01:23.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: mon.a calling monitor election 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: monmap epoch 3 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: last_changed 2024-04-26T20:01:14.970806+0000 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: created 2024-04-26T19:58:53.653581+0000 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: min_mon_release 19 (squid) 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: election_strategy: 1 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: 0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: fsmap 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: mgrmap e13: x(active, since 70s) 2024-04-26T20:01:23.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: overall HEALTH_OK 2024-04-26T20:01:23.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.279+0000 7f4988800700 1 -- 172.21.15.94:0/3759200826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f498006a840 msgr2=0x7f498006ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:23.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.279+0000 7f4988800700 1 --2- 172.21.15.94:0/3759200826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f498006a840 0x7f498006ac20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4974009910 tx=0x7f4974009c20 comp rx=0 tx=0).stop 2024-04-26T20:01:23.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.279+0000 7f4988800700 1 -- 172.21.15.94:0/3759200826 shutdown_connections 2024-04-26T20:01:23.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.279+0000 7f4988800700 1 --2- 172.21.15.94:0/3759200826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f498006a840 0x7f498006ac20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.279+0000 7f4988800700 1 -- 172.21.15.94:0/3759200826 >> 172.21.15.94:0/3759200826 conn(0x7f4980077880 msgr2=0x7f4980077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.279+0000 7f4988800700 1 -- 172.21.15.94:0/3759200826 shutdown_connections 2024-04-26T20:01:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.280+0000 7f4988800700 1 -- 172.21.15.94:0/3759200826 wait complete. 2024-04-26T20:01:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.281+0000 7f4988800700 1 Processor -- start 2024-04-26T20:01:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.281+0000 7f4988800700 1 -- start start 2024-04-26T20:01:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4988800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f498006a840 0x7f498019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4988800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f498019b280 0x7f498019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4988800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f498019fb50 0x7f49801a0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4988800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f498010f570 con 0x7f498019fb50 2024-04-26T20:01:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4988800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f498010f3f0 con 0x7f498019b280 2024-04-26T20:01:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4988800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f498010f6f0 con 0x7f498006a840 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4986800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f498019fb50 0x7f49801a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4985e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f498006a840 0x7f498019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4985e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f498006a840 0x7f498019ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:49728/0 (socket says 172.21.15.94:49728) 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4985e00700 1 -- 172.21.15.94:0/554697550 learned_addr learned my addr 172.21.15.94:0/554697550 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4985400700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f498019b280 0x7f498019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4985e00700 1 -- 172.21.15.94:0/554697550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f498019b280 msgr2=0x7f498019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4985e00700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f498019b280 0x7f498019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4985e00700 1 -- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f498019fb50 msgr2=0x7f49801a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.282+0000 7f4985e00700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f498019fb50 0x7f49801a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.283+0000 7f4985e00700 1 -- 172.21.15.94:0/554697550 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49801a78c0 con 0x7f498006a840 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.283+0000 7f4986800700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f498019fb50 0x7f49801a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:01:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.283+0000 7f4985e00700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f498006a840 0x7f498019ad40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4974004f80 tx=0x7f4974004fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.283+0000 7f497aa00700 1 -- 172.21.15.94:0/554697550 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974026070 con 0x7f498006a840 2024-04-26T20:01:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.283+0000 7f497aa00700 1 -- 172.21.15.94:0/554697550 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f497400b750 con 0x7f498006a840 2024-04-26T20:01:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.283+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49801a7a90 con 0x7f498006a840 2024-04-26T20:01:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.283+0000 7f497aa00700 1 -- 172.21.15.94:0/554697550 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974009e60 con 0x7f498006a840 2024-04-26T20:01:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.283+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49801a7fc0 con 0x7f498006a840 2024-04-26T20:01:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.284+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4980064a80 con 0x7f498006a840 2024-04-26T20:01:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.285+0000 7f497aa00700 1 -- 172.21.15.94:0/554697550 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4974029070 con 0x7f498006a840 2024-04-26T20:01:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.285+0000 7f497aa00700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4968039ae0 0x7f496803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.285+0000 7f497aa00700 1 -- 172.21.15.94:0/554697550 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f497404d3b0 con 0x7f498006a840 2024-04-26T20:01:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.286+0000 7f4985400700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4968039ae0 0x7f496803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.286+0000 7f4985400700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4968039ae0 0x7f496803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4970009ad0 tx=0x7f4970009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.291+0000 7f497aa00700 1 -- 172.21.15.94:0/554697550 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4974013070 con 0x7f498006a840 2024-04-26T20:01:23.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.521+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f498019c210 con 0x7f498006a840 2024-04-26T20:01:23.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.521+0000 7f497aa00700 1 -- 172.21.15.94:0/554697550 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f498019c210 con 0x7f498006a840 2024-04-26T20:01:23.523 INFO:teuthology.orchestra.run.smithi094.stdout:# minimal ceph.conf for 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:01:23.523 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-04-26T20:01:23.523 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:01:23.523 INFO:teuthology.orchestra.run.smithi094.stdout: mon_host = [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-26T20:01:23.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.524+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4968039ae0 msgr2=0x7f496803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:23.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.524+0000 7f4988800700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4968039ae0 0x7f496803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4970009ad0 tx=0x7f4970009210 comp rx=0 tx=0).stop 2024-04-26T20:01:23.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.524+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f498006a840 msgr2=0x7f498019ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:23.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.524+0000 7f4988800700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f498006a840 0x7f498019ad40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4974004f80 tx=0x7f4974004fb0 comp rx=0 tx=0).stop 2024-04-26T20:01:23.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.525+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 shutdown_connections 2024-04-26T20:01:23.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.525+0000 7f4988800700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f498019fb50 0x7f49801a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:23.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.525+0000 7f4988800700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f498006a840 0x7f498019ad40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.525+0000 7f4988800700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4968039ae0 0x7f496803bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.525+0000 7f4988800700 1 --2- 172.21.15.94:0/554697550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f498019b280 0x7f498019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.525+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 >> 172.21.15.94:0/554697550 conn(0x7f4980077880 msgr2=0x7f4980100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.525+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 shutdown_connections 2024-04-26T20:01:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:23.525+0000 7f4988800700 1 -- 172.21.15.94:0/554697550 wait complete. 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: Updating smithi204:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: Updating smithi103:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: Updating smithi094:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.871 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:23.872 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:01:23.872 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:23.872 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/554697550' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:23.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: Updating smithi204:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: Updating smithi103:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: Updating smithi094:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:01:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:23.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/554697550' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:24.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: Updating smithi204:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: Updating smithi103:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: Updating smithi094:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/554697550' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:24.248 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T20:01:24.248 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:01:24.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T20:01:24.323 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:01:24.323 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:01:24.445 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T20:01:24.445 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T20:01:24.472 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T20:01:24.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:01:24.537 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T20:01:24.537 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T20:01:24.564 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T20:01:24.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:01:24.627 INFO:tasks.cephadm:Adding mgr.x on smithi094 2024-04-26T20:01:24.627 INFO:tasks.cephadm:Adding mgr.y on smithi103 2024-04-26T20:01:24.627 INFO:tasks.cephadm:Adding mgr.z on smithi204 2024-04-26T20:01:24.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi094=x;smithi103=y;smithi204=z' 2024-04-26T20:01:24.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:24 smithi094 ceph-mon[28050]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T20:01:24.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:24 smithi094 ceph-mon[28050]: Reconfiguring daemon mon.a on smithi094 2024-04-26T20:01:24.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:24 smithi094 ceph-mon[28050]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:24.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:24 smithi103 ceph-mon[32635]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T20:01:24.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:24 smithi103 ceph-mon[32635]: Reconfiguring daemon mon.a on smithi094 2024-04-26T20:01:24.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:24 smithi103 ceph-mon[32635]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:24.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:24 smithi204 ceph-mon[32618]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T20:01:24.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:24 smithi204 ceph-mon[32618]: Reconfiguring daemon mon.a on smithi094 2024-04-26T20:01:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:24 smithi204 ceph-mon[32618]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:24.898 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:01:26.409 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.406+0000 7f26901ac700 1 -- 172.21.15.204:0/3563736179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2688103a30 msgr2=0x7f2688105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.406+0000 7f26901ac700 1 --2- 172.21.15.204:0/3563736179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2688103a30 0x7f2688105e20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2684009910 tx=0x7f2684009c20 comp rx=0 tx=0).stop 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.406+0000 7f26901ac700 1 -- 172.21.15.204:0/3563736179 shutdown_connections 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.406+0000 7f26901ac700 1 --2- 172.21.15.204:0/3563736179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2688103a30 0x7f2688105e20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.406+0000 7f26901ac700 1 -- 172.21.15.204:0/3563736179 >> 172.21.15.204:0/3563736179 conn(0x7f26880fd460 msgr2=0x7f26880ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.406+0000 7f26901ac700 1 -- 172.21.15.204:0/3563736179 shutdown_connections 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.406+0000 7f26901ac700 1 -- 172.21.15.204:0/3563736179 wait complete. 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f26901ac700 1 Processor -- start 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f26901ac700 1 -- start start 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f26901ac700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2688103a30 0x7f2688198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f26901ac700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2688198ba0 0x7f268819dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:26.410 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f26901ac700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2688199020 0x7f26881a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f26901ac700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2688108e80 con 0x7f2688198ba0 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f26901ac700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2688108d00 con 0x7f2688199020 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f26901ac700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2688109000 con 0x7f2688103a30 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f268df48700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2688103a30 0x7f2688198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f268df48700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2688103a30 0x7f2688198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:34022/0 (socket says 172.21.15.204:34022) 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.407+0000 7f268df48700 1 -- 172.21.15.204:0/1469153139 learned_addr learned my addr 172.21.15.204:0/1469153139 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f268df48700 1 -- 172.21.15.204:0/1469153139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2688199020 msgr2=0x7f26881a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f268df48700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2688199020 0x7f26881a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f268df48700 1 -- 172.21.15.204:0/1469153139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2688198ba0 msgr2=0x7f268819dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f268df48700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2688198ba0 0x7f268819dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f268df48700 1 -- 172.21.15.204:0/1469153139 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26881a0a70 con 0x7f2688103a30 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f268df48700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2688103a30 0x7f2688198660 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2684005e00 tx=0x7f2684005050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:26.411 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f267effd700 1 -- 172.21.15.204:0/1469153139 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684026070 con 0x7f2688103a30 2024-04-26T20:01:26.412 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f267effd700 1 -- 172.21.15.204:0/1469153139 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268400be10 con 0x7f2688103a30 2024-04-26T20:01:26.412 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2688101880 con 0x7f2688103a30 2024-04-26T20:01:26.412 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f267effd700 1 -- 172.21.15.204:0/1469153139 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684017440 con 0x7f2688103a30 2024-04-26T20:01:26.412 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.408+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2688101e40 con 0x7f2688103a30 2024-04-26T20:01:26.413 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.409+0000 7f267effd700 1 -- 172.21.15.204:0/1469153139 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2684029070 con 0x7f2688103a30 2024-04-26T20:01:26.413 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.409+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2688064a80 con 0x7f2688103a30 2024-04-26T20:01:26.413 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.409+0000 7f267effd700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2674039ae0 0x7f267403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:26.414 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.409+0000 7f267effd700 1 -- 172.21.15.204:0/1469153139 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f268404d800 con 0x7f2688103a30 2024-04-26T20:01:26.414 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.410+0000 7f268d747700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2674039ae0 0x7f267403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:26.414 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.411+0000 7f268d747700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2674039ae0 0x7f267403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26780098e0 tx=0x7f2678006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:26.415 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.414+0000 7f267effd700 1 -- 172.21.15.204:0/1469153139 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2684013070 con 0x7f2688103a30 2024-04-26T20:01:26.626 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.624+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi094=x;smithi103=y;smithi204=z", "target": ["mon-mgr", ""]}) -- 0x7f2688103320 con 0x7f2674039ae0 2024-04-26T20:01:26.626 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:26 smithi204 ceph-mon[32618]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:26.626 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:26.627 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:26.627 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:26.627 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:01:26.627 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:26.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.638+0000 7f267effd700 1 -- 172.21.15.204:0/1469153139 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2688103320 con 0x7f2674039ae0 2024-04-26T20:01:26.639 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-04-26T20:01:26.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2674039ae0 msgr2=0x7f267403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:26.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2674039ae0 0x7f267403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26780098e0 tx=0x7f2678006dd0 comp rx=0 tx=0).stop 2024-04-26T20:01:26.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2688103a30 msgr2=0x7f2688198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:26.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2688103a30 0x7f2688198660 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2684005e00 tx=0x7f2684005050 comp rx=0 tx=0).stop 2024-04-26T20:01:26.642 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 shutdown_connections 2024-04-26T20:01:26.642 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2674039ae0 0x7f267403bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:26.642 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2688199020 0x7f26881a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:26.642 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2688103a30 0x7f2688198660 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:26.642 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 --2- 172.21.15.204:0/1469153139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2688198ba0 0x7f268819dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:26.642 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 >> 172.21.15.204:0/1469153139 conn(0x7f26880fd460 msgr2=0x7f26880fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:26.642 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 shutdown_connections 2024-04-26T20:01:26.642 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:01:26.640+0000 7f26901ac700 1 -- 172.21.15.204:0/1469153139 wait complete. 2024-04-26T20:01:26.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:26 smithi094 ceph-mon[28050]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:26.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:26.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:26.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:26.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:01:26.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:26 smithi103 ceph-mon[32635]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:27.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:27.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:01:27.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:27.203 DEBUG:teuthology.orchestra.run.smithi103:mgr.y> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T20:01:27.206 DEBUG:teuthology.orchestra.run.smithi204:mgr.z> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T20:01:27.208 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T20:01:27.209 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:01:27.209 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T20:01:27.266 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T20:01:27.266 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-04-26T20:01:27.316 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T20:01:27.316 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:27.316 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-26T20:01:27.316 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:27.316 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:27.317 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-26 20:00:24.492428186 +0000 2024-04-26T20:01:27.317 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-26 19:54:43.421130026 +0000 2024-04-26T20:01:27.317 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-26 19:54:43.421130026 +0000 2024-04-26T20:01:27.317 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-26 19:54:43.421130026 +0000 2024-04-26T20:01:27.317 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T20:01:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-26T20:01:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-26T20:01:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000270654 s, 1.9 MB/s 2024-04-26T20:01:27.393 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T20:01:27.449 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-04-26T20:01:27.499 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T20:01:27.499 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:27.499 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-04-26T20:01:27.499 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:27.499 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:27.499 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-26 20:00:24.547427251 +0000 2024-04-26T20:01:27.499 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-26 19:54:44.017119652 +0000 2024-04-26T20:01:27.499 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-26 19:54:44.017119652 +0000 2024-04-26T20:01:27.499 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-26 19:54:44.017119652 +0000 2024-04-26T20:01:27.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T20:01:27.576 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-26T20:01:27.576 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-26T20:01:27.576 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000265035 s, 1.9 MB/s 2024-04-26T20:01:27.580 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T20:01:27.635 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-04-26T20:01:27.683 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T20:01:27.683 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:27.683 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-04-26T20:01:27.683 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:27.683 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:27.683 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-26 20:00:24.601426334 +0000 2024-04-26T20:01:27.683 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-26 19:54:44.618109192 +0000 2024-04-26T20:01:27.683 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-26 19:54:44.618109192 +0000 2024-04-26T20:01:27.684 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-26 19:54:44.618109192 +0000 2024-04-26T20:01:27.684 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: Reconfiguring mon.b (monmap changed)... 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: Reconfiguring daemon mon.b on smithi103 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi094=x;smithi103=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: Saving service mgr spec with placement smithi094=x;smithi103=y;smithi204=z;count:3 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:01:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:27.751 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-26T20:01:27.751 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-26T20:01:27.751 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000246989 s, 2.1 MB/s 2024-04-26T20:01:27.754 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T20:01:27.801 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: Reconfiguring mon.b (monmap changed)... 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: Reconfiguring daemon mon.b on smithi103 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi094=x;smithi103=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: Saving service mgr spec with placement smithi094=x;smithi103=y;smithi204=z;count:3 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:01:27.848 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:27.849 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T20:01:27.849 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:27.849 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-26T20:01:27.849 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:27.850 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:27.850 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-04-26 20:00:24.656425399 +0000 2024-04-26T20:01:27.850 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-04-26 19:54:45.227098592 +0000 2024-04-26T20:01:27.850 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-04-26 19:54:45.227098592 +0000 2024-04-26T20:01:27.850 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-04-26 19:54:45.227098592 +0000 2024-04-26T20:01:27.850 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T20:01:27.916 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-04-26T20:01:27.916 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-04-26T20:01:27.916 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000226973 s, 2.3 MB/s 2024-04-26T20:01:27.919 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T20:01:27.974 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T20:01:27.974 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T20:01:27.990 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T20:01:27.991 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-04-26T20:01:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: Reconfiguring mon.b (monmap changed)... 2024-04-26T20:01:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: Reconfiguring daemon mon.b on smithi103 2024-04-26T20:01:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi094=x;smithi103=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:01:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: Saving service mgr spec with placement smithi094=x;smithi103=y;smithi204=z;count:3 2024-04-26T20:01:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:28.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:01:28.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:01:28.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:28.046 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T20:01:28.046 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:28.046 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-26T20:01:28.046 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:28.046 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:28.046 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 20:01:14.335039732 +0000 2024-04-26T20:01:28.046 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 19:53:39.584477557 +0000 2024-04-26T20:01:28.046 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 19:53:39.584477557 +0000 2024-04-26T20:01:28.047 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 19:53:39.584477557 +0000 2024-04-26T20:01:28.047 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T20:01:28.109 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T20:01:28.109 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T20:01:28.109 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000212429 s, 2.4 MB/s 2024-04-26T20:01:28.110 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T20:01:28.166 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-04-26T20:01:28.222 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T20:01:28.222 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:28.222 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-26T20:01:28.222 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:28.222 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:28.222 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 20:01:14.336039714 +0000 2024-04-26T20:01:28.222 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 19:53:39.935470499 +0000 2024-04-26T20:01:28.222 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 19:53:39.935470499 +0000 2024-04-26T20:01:28.222 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 19:53:39.935470499 +0000 2024-04-26T20:01:28.223 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T20:01:28.283 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T20:01:28.284 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T20:01:28.284 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000204757 s, 2.5 MB/s 2024-04-26T20:01:28.285 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T20:01:28.341 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-04-26T20:01:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T20:01:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:28.395 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-26T20:01:28.395 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:28.396 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:28.396 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 20:01:14.336039714 +0000 2024-04-26T20:01:28.396 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 19:53:40.281463541 +0000 2024-04-26T20:01:28.396 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 19:53:40.281463541 +0000 2024-04-26T20:01:28.396 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 19:53:40.281463541 +0000 2024-04-26T20:01:28.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T20:01:28.457 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T20:01:28.458 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T20:01:28.458 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000184946 s, 2.8 MB/s 2024-04-26T20:01:28.459 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T20:01:28.514 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-04-26T20:01:28.568 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T20:01:28.568 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:28.568 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-26T20:01:28.568 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:28.568 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:28.568 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 20:01:14.336039714 +0000 2024-04-26T20:01:28.568 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 19:53:40.616456804 +0000 2024-04-26T20:01:28.568 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 19:53:40.616456804 +0000 2024-04-26T20:01:28.569 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 19:53:40.616456804 +0000 2024-04-26T20:01:28.569 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T20:01:28.631 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T20:01:28.631 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T20:01:28.631 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000196836 s, 2.6 MB/s 2024-04-26T20:01:28.632 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T20:01:28.688 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-26T20:01:28.688 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T20:01:28.704 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T20:01:28.704 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-04-26T20:01:28.760 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T20:01:28.760 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:28.760 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-26T20:01:28.760 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:28.760 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:28.760 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-26 20:01:05.374329472 +0000 2024-04-26T20:01:28.760 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-26 19:53:45.254837334 +0000 2024-04-26T20:01:28.760 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-26 19:53:45.254837334 +0000 2024-04-26T20:01:28.760 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-04-26 19:53:45.254837334 +0000 2024-04-26T20:01:28.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T20:01:28.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:28 smithi103 ceph-mon[32635]: Reconfiguring mon.c (monmap changed)... 2024-04-26T20:01:28.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:28 smithi103 ceph-mon[32635]: Reconfiguring daemon mon.c on smithi204 2024-04-26T20:01:28.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:28 smithi103 ceph-mon[32635]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-26T20:01:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-26T20:01:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000273331 s, 1.9 MB/s 2024-04-26T20:01:28.823 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T20:01:28.879 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-04-26T20:01:28.925 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:28 smithi204 ceph-mon[32618]: Reconfiguring mon.c (monmap changed)... 2024-04-26T20:01:28.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:28 smithi204 ceph-mon[32618]: Reconfiguring daemon mon.c on smithi204 2024-04-26T20:01:28.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:28 smithi204 ceph-mon[32618]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:28.940 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T20:01:28.940 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:28.940 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-26T20:01:28.940 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:28.940 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:28.941 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-26 20:01:05.374329472 +0000 2024-04-26T20:01:28.941 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-26 19:53:45.601831902 +0000 2024-04-26T20:01:28.941 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-26 19:53:45.601831902 +0000 2024-04-26T20:01:28.941 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-04-26 19:53:45.601831902 +0000 2024-04-26T20:01:28.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T20:01:28.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:28 smithi094 ceph-mon[28050]: Reconfiguring mon.c (monmap changed)... 2024-04-26T20:01:28.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:28 smithi094 ceph-mon[28050]: Reconfiguring daemon mon.c on smithi204 2024-04-26T20:01:28.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:28 smithi094 ceph-mon[28050]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:29.003 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-26T20:01:29.003 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-26T20:01:29.003 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000198665 s, 2.6 MB/s 2024-04-26T20:01:29.004 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T20:01:29.060 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-04-26T20:01:29.115 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T20:01:29.115 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:29.115 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-26T20:01:29.115 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:29.115 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:29.116 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-26 20:01:05.375329459 +0000 2024-04-26T20:01:29.116 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-26 19:53:45.935826673 +0000 2024-04-26T20:01:29.116 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-26 19:53:45.935826673 +0000 2024-04-26T20:01:29.116 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-04-26 19:53:45.935826673 +0000 2024-04-26T20:01:29.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T20:01:29.180 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-26T20:01:29.181 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-26T20:01:29.181 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000250058 s, 2.0 MB/s 2024-04-26T20:01:29.182 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T20:01:29.239 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-04-26T20:01:29.294 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T20:01:29.295 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:01:29.295 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-26T20:01:29.295 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:01:29.295 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:01:29.295 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-26 20:01:05.375329459 +0000 2024-04-26T20:01:29.295 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-26 19:53:46.266821490 +0000 2024-04-26T20:01:29.295 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-26 19:53:46.266821490 +0000 2024-04-26T20:01:29.295 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-04-26 19:53:46.266821490 +0000 2024-04-26T20:01:29.295 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T20:01:29.356 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-26T20:01:29.356 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-26T20:01:29.357 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000174888 s, 2.9 MB/s 2024-04-26T20:01:29.358 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T20:01:29.413 INFO:tasks.cephadm:Deploying osd.0 on smithi094 with /dev/vg_nvme/lv_4... 2024-04-26T20:01:29.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T20:01:29.835 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:01:29.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:29.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:29.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:29.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:29.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:29.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:29.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:01:29.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:01:29.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:01:29.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:29.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: Deploying daemon mgr.z on smithi204 2024-04-26T20:01:29.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:29 smithi094 ceph-mon[28050]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: Deploying daemon mgr.z on smithi204 2024-04-26T20:01:30.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:29 smithi204 ceph-mon[32618]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:30.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:30.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:30.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:30.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:30.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:30.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:30.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:01:30.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:01:30.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:01:30.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:30.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: Deploying daemon mgr.z on smithi204 2024-04-26T20:01:30.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:29 smithi103 ceph-mon[32635]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:30.882 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:30 smithi204 systemd[1]: Starting Ceph mgr.z for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:01:31.259 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:30 smithi204 podman[34454]: 2024-04-26 20:01:30.938997036 +0000 UTC m=+0.111611420 container create aa57009e2a3cd344407767d22ee94fe57d4ed722ba251ed3c03773ef77e0ebd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-26T20:01:31.259 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:30 smithi204 podman[34454]: 2024-04-26 20:01:30.845416823 +0000 UTC m=+0.018031204 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:01:31.259 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 podman[34454]: 2024-04-26 20:01:31.152280925 +0000 UTC m=+0.324895311 container init aa57009e2a3cd344407767d22ee94fe57d4ed722ba251ed3c03773ef77e0ebd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T20:01:31.259 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 podman[34454]: 2024-04-26 20:01:31.154841753 +0000 UTC m=+0.327456134 container start aa57009e2a3cd344407767d22ee94fe57d4ed722ba251ed3c03773ef77e0ebd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-26T20:01:31.259 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 ceph-mgr[34473]: -- 172.21.15.204:0/2770433644 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607cd55a5a0 con 0x5607cd52f000 2024-04-26T20:01:31.259 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 bash[34454]: aa57009e2a3cd344407767d22ee94fe57d4ed722ba251ed3c03773ef77e0ebd0 2024-04-26T20:01:31.597 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 systemd[1]: Started Ceph mgr.z for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:01:31.597 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:31.305+0000 7f547287f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T20:01:31.597 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:31.493+0000 7f547287f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:01:31.597 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:31.594+0000 7f547287f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T20:01:32.331 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:31.903+0000 7f547287f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T20:01:32.331 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:31.987+0000 7f547287f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T20:01:32.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:32.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:32.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:32.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:01:32.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:01:32.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:01:32.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:32.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:32 smithi204 ceph-mon[32618]: Deploying daemon mgr.y on smithi103 2024-04-26T20:01:32.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:32 smithi204 ceph-mon[32618]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:32.693 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T20:01:32.693 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T20:01:32.693 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-04-26T20:01:32.693 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-04-26T20:01:32.693 INFO:teuthology.orchestra.run.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0363378 s, 289 MB/s 2024-04-26T20:01:32.693 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-04-26T20:01:32.693 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:32.693 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:32.693 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:32.694 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:01:32.694 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:01:32.694 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:01:32.694 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:32.694 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:32 smithi094 ceph-mon[28050]: Deploying daemon mgr.y on smithi103 2024-04-26T20:01:32.694 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:32 smithi094 ceph-mon[28050]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:32.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:32.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:32.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:32.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:01:32.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:01:32.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:01:32.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:32.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:32 smithi103 ceph-mon[32635]: Deploying daemon mgr.y on smithi103 2024-04-26T20:01:32.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:32 smithi103 ceph-mon[32635]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:33.407 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_4 2024-04-26T20:01:33.535 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:33 smithi103 systemd[1]: Starting Ceph mgr.y for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:01:33.536 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:33.189+0000 7f547287f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T20:01:33.536 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:33.360+0000 7f547287f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T20:01:33.536 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:33.534+0000 7f547287f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T20:01:33.826 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:33 smithi103 podman[34001]: 2024-04-26 20:01:33.463992861 +0000 UTC m=+0.017707601 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:01:33.826 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:33 smithi103 podman[34001]: 2024-04-26 20:01:33.573719114 +0000 UTC m=+0.127433849 container create d8d8492861488bdd01b4dd911f841ccc8ad1187e2565523789984b04919362ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:01:33.826 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:33 smithi103 podman[34001]: 2024-04-26 20:01:33.776016708 +0000 UTC m=+0.329731441 container init d8d8492861488bdd01b4dd911f841ccc8ad1187e2565523789984b04919362ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-26T20:01:33.827 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:33 smithi103 podman[34001]: 2024-04-26 20:01:33.778365522 +0000 UTC m=+0.332080254 container start d8d8492861488bdd01b4dd911f841ccc8ad1187e2565523789984b04919362ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:01:33.827 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:33 smithi103 ceph-mgr[34021]: -- 172.21.15.103:0/2097425810 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c8108c65a0 con 0x55c81089a800 2024-04-26T20:01:33.834 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:01:34.121 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:33 smithi103 bash[34001]: d8d8492861488bdd01b4dd911f841ccc8ad1187e2565523789984b04919362ec 2024-04-26T20:01:34.121 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:33 smithi103 systemd[1]: Started Ceph mgr.y for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:01:34.121 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:33 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:33.933+0000 7f2951481200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T20:01:34.122 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:34.120+0000 7f2951481200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:01:34.122 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:33.818+0000 7f547287f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T20:01:34.122 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:33.914+0000 7f547287f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T20:01:34.122 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:34 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:34.121+0000 7f547287f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T20:01:34.381 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:34 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:34.319+0000 7f547287f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:01:34.496 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:34.223+0000 7f2951481200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T20:01:34.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:34 smithi094 ceph-mon[28050]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:34.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:34 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:34 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:34 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:34 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:34 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:34.785 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:34.549+0000 7f2951481200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T20:01:34.785 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:34.635+0000 7f2951481200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T20:01:34.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-mon[32635]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:34.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:34 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:34.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:34 smithi204 ceph-mon[32618]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:34.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:34 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:34 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:34 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:34 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:34.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:34 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:35.382 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:34 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:34.972+0000 7f547287f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T20:01:35.717 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:35 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:35.406+0000 7f547287f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T20:01:35.717 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:35 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:35.507+0000 7f547287f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T20:01:35.717 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:35 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:35.716+0000 7f547287f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T20:01:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.888+0000 7f022aa00700 1 -- 172.21.15.94:0/37384682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f022410acc0 msgr2=0x7f022410b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.888+0000 7f022aa00700 1 --2- 172.21.15.94:0/37384682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f022410acc0 0x7f022410b120 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f02100099c0 tx=0x7f0210009cd0 comp rx=0 tx=0).stop 2024-04-26T20:01:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.888+0000 7f022aa00700 1 -- 172.21.15.94:0/37384682 shutdown_connections 2024-04-26T20:01:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.888+0000 7f022aa00700 1 --2- 172.21.15.94:0/37384682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022410b660 0x7f0224111f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.888+0000 7f022aa00700 1 --2- 172.21.15.94:0/37384682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0224109ac0 0x7f0224109ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.888+0000 7f022aa00700 1 --2- 172.21.15.94:0/37384682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f022410acc0 0x7f022410b120 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.888+0000 7f022aa00700 1 -- 172.21.15.94:0/37384682 >> 172.21.15.94:0/37384682 conn(0x7f0224078a90 msgr2=0x7f022407aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:01:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.888+0000 7f022aa00700 1 -- 172.21.15.94:0/37384682 shutdown_connections 2024-04-26T20:01:35.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.888+0000 7f022aa00700 1 -- 172.21.15.94:0/37384682 wait complete. 2024-04-26T20:01:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.889+0000 7f022aa00700 1 Processor -- start 2024-04-26T20:01:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.889+0000 7f022aa00700 1 -- start start 2024-04-26T20:01:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f022aa00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0224109ac0 0x7f022419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f022aa00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022410acc0 0x7f02241a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f022aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f022410b660 0x7f02241a7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f022aa00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0224115580 con 0x7f0224109ac0 2024-04-26T20:01:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f022aa00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0224115400 con 0x7f022410b660 2024-04-26T20:01:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f022aa00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0224115700 con 0x7f022410acc0 2024-04-26T20:01:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0223e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0224109ac0 0x7f022419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0228a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f022410b660 0x7f02241a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0223400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022410acc0 0x7f02241a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0223e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0224109ac0 0x7f022419fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52274/0 (socket says 172.21.15.94:52274) 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0223e00700 1 -- 172.21.15.94:0/1857357905 learned_addr learned my addr 172.21.15.94:0/1857357905 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0228a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f022410b660 0x7f02241a7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:59552/0 (socket says 172.21.15.94:59552) 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0228a00700 1 -- 172.21.15.94:0/1857357905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022410acc0 msgr2=0x7f02241a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0228a00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022410acc0 0x7f02241a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0228a00700 1 -- 172.21.15.94:0/1857357905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0224109ac0 msgr2=0x7f022419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0228a00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0224109ac0 0x7f022419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.890+0000 7f0228a00700 1 -- 172.21.15.94:0/1857357905 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02241a7830 con 0x7f022410b660 2024-04-26T20:01:35.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.891+0000 7f0223400700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022410acc0 0x7f02241a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:01:35.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.891+0000 7f0228a00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f022410b660 0x7f02241a7190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f021c00efb0 tx=0x7f021c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:35.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.891+0000 7f0220c00700 1 -- 172.21.15.94:0/1857357905 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021c003a00 con 0x7f022410b660 2024-04-26T20:01:35.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.891+0000 7f0220c00700 1 -- 172.21.15.94:0/1857357905 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f021c00ce50 con 0x7f022410b660 2024-04-26T20:01:35.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.891+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02241a7b20 con 0x7f022410b660 2024-04-26T20:01:35.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.891+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02241a8060 con 0x7f022410b660 2024-04-26T20:01:35.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.891+0000 7f0220c00700 1 -- 172.21.15.94:0/1857357905 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021c0047b0 con 0x7f022410b660 2024-04-26T20:01:35.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.892+0000 7f0220c00700 1 -- 172.21.15.94:0/1857357905 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f021c012050 con 0x7f022410b660 2024-04-26T20:01:35.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.892+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f022406a110 con 0x7f022410b660 2024-04-26T20:01:35.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.893+0000 7f0220c00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0208039890 0x7f020803bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:01:35.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.893+0000 7f0220c00700 1 -- 172.21.15.94:0/1857357905 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f021c04c470 con 0x7f022410b660 2024-04-26T20:01:35.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.893+0000 7f0223e00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0208039890 0x7f020803bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:01:35.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.894+0000 7f0223e00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0208039890 0x7f020803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0214009680 tx=0x7f0214009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:01:35.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:35.898+0000 7f0220c00700 1 -- 172.21.15.94:0/1857357905 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f021c022060 con 0x7f022410b660 2024-04-26T20:01:36.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:36.138+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0224063860 con 0x7f0208039890 2024-04-26T20:01:36.227 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:35 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:35.888+0000 7f2951481200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T20:01:36.227 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:36.056+0000 7f2951481200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T20:01:36.227 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:36.226+0000 7f2951481200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T20:01:36.507 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:36.245+0000 7f547287f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T20:01:36.507 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:36.337+0000 7f547287f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T20:01:36.784 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:36.512+0000 7f2951481200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T20:01:36.784 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:36.605+0000 7f2951481200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T20:01:36.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:01:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:01:36.786 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:36.787 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:01:36.787 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:01:36.787 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:36.808 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:36.579+0000 7f547287f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T20:01:36.808 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:36.711+0000 7f547287f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:36.807+0000 7f547287f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:01:36.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:01:36.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:37.131 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:36.923+0000 7f547287f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T20:01:37.284 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:36.806+0000 7f2951481200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T20:01:37.284 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:36 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:36.995+0000 7f2951481200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:01:37.520 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:37.228+0000 7f547287f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T20:01:37.784 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:37 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:37.592+0000 7f2951481200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T20:01:37.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:37 smithi103 ceph-mon[32635]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T20:01:37.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:37 smithi103 ceph-mon[32635]: Reconfiguring daemon mgr.x on smithi094 2024-04-26T20:01:37.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:37 smithi103 ceph-mon[32635]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:01:37.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:37 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:37.831 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:37 smithi204 ceph-mon[32618]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T20:01:37.831 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:37 smithi204 ceph-mon[32618]: Reconfiguring daemon mgr.x on smithi094 2024-04-26T20:01:37.832 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:37 smithi204 ceph-mon[32618]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:01:37.832 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:37 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:37.832 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:37 smithi094 ceph-mon[28050]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T20:01:37.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:37 smithi094 ceph-mon[28050]: Reconfiguring daemon mgr.x on smithi094 2024-04-26T20:01:37.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:37 smithi094 ceph-mon[28050]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:01:37.833 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:37 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.132 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:37.830+0000 7f547287f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T20:01:38.132 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z[34469]: 2024-04-26T20:01:38.044+0000 7f547287f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T20:01:38.284 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:38.014+0000 7f2951481200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T20:01:38.284 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:38.111+0000 7f2951481200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T20:01:38.784 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:38.314+0000 7f2951481200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: Standby manager daemon z started 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T20:01:38.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:38 smithi094 ceph-mon[28050]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: Standby manager daemon z started 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T20:01:38.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-mon[32635]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:01:38.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: Standby manager daemon z started 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T20:01:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:38 smithi204 ceph-mon[32618]: from='mgr.? 172.21.15.204:0/1992358067' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:01:39.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:39.031+0000 7f0220c00700 1 -- 172.21.15.94:0/1857357905 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f021c014480 con 0x7f022410b660 2024-04-26T20:01:39.169 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:38.834+0000 7f2951481200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T20:01:39.169 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:38.925+0000 7f2951481200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T20:01:39.513 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:39.168+0000 7f2951481200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T20:01:39.513 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:39.302+0000 7f2951481200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T20:01:39.513 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:39.397+0000 7f2951481200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T20:01:39.513 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:39.512+0000 7f2951481200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T20:01:40.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:40 smithi103 ceph-mon[32635]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-26T20:01:40.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T20:01:40.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:40 smithi103 ceph-mon[32635]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:40.284 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:39.810+0000 7f2951481200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T20:01:40.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:40 smithi204 ceph-mon[32618]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-26T20:01:40.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T20:01:40.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:40 smithi204 ceph-mon[32618]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:40.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:40 smithi094 ceph-mon[28050]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-26T20:01:40.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T20:01:40.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:40 smithi094 ceph-mon[28050]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:40.784 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:40.409+0000 7f2951481200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T20:01:40.784 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:01:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y[34017]: 2024-04-26T20:01:40.628+0000 7f2951481200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T20:01:41.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:01:41.057+0000 7f0220c00700 1 -- 172.21.15.94:0/1857357905 <== mon.2 v2:172.21.15.103:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f021c013ce0 con 0x7f022410b660 2024-04-26T20:01:41.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: Standby manager daemon y started 2024-04-26T20:01:41.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T20:01:41.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:01:41.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T20:01:41.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:01:41.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/126525760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d79ff6-dda1-4462-b2e6-eac0c688fef7"}]: dispatch 2024-04-26T20:01:41.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d79ff6-dda1-4462-b2e6-eac0c688fef7"}]: dispatch 2024-04-26T20:01:41.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6d79ff6-dda1-4462-b2e6-eac0c688fef7"}]': finished 2024-04-26T20:01:41.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T20:01:41.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:41 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: Standby manager daemon y started 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/126525760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d79ff6-dda1-4462-b2e6-eac0c688fef7"}]: dispatch 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d79ff6-dda1-4462-b2e6-eac0c688fef7"}]: dispatch 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6d79ff6-dda1-4462-b2e6-eac0c688fef7"}]': finished 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T20:01:41.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:41 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:01:41.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: Standby manager daemon y started 2024-04-26T20:01:41.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T20:01:41.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:01:41.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T20:01:41.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: from='mgr.? 172.21.15.103:0/883896993' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:01:41.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/126525760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d79ff6-dda1-4462-b2e6-eac0c688fef7"}]: dispatch 2024-04-26T20:01:41.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6d79ff6-dda1-4462-b2e6-eac0c688fef7"}]: dispatch 2024-04-26T20:01:41.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6d79ff6-dda1-4462-b2e6-eac0c688fef7"}]': finished 2024-04-26T20:01:41.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T20:01:41.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:41 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:01:42.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:42 smithi103 ceph-mon[32635]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-26T20:01:42.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T20:01:42.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:42 smithi103 ceph-mon[32635]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:42.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:42 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3166039859' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:01:42.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:42 smithi204 ceph-mon[32618]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-26T20:01:42.375 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T20:01:42.375 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:42 smithi204 ceph-mon[32618]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:42.375 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:42 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3166039859' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:01:42.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:42 smithi094 ceph-mon[28050]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-26T20:01:42.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T20:01:42.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:42 smithi094 ceph-mon[28050]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:42.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:42 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3166039859' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:01:44.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:44 smithi103 ceph-mon[32635]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:44 smithi204 ceph-mon[32618]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:44.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:44 smithi094 ceph-mon[28050]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:46.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:46 smithi103 ceph-mon[32635]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:46.866 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:46 smithi094 ceph-mon[28050]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:46.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:46 smithi204 ceph-mon[32618]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:48.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:48 smithi094 ceph-mon[28050]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:48.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:48 smithi103 ceph-mon[32635]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:48.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:48 smithi204 ceph-mon[32618]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:50.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:50 smithi103 ceph-mon[32635]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:50.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T20:01:50.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:50.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:50 smithi204 ceph-mon[32618]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:50.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T20:01:50.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:50.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:50 smithi094 ceph-mon[28050]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:50.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T20:01:50.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:01:51.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:51 smithi103 ceph-mon[32635]: Deploying daemon osd.0 on smithi094 2024-04-26T20:01:51.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:51 smithi204 ceph-mon[32618]: Deploying daemon osd.0 on smithi094 2024-04-26T20:01:51.922 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:51 smithi094 ceph-mon[28050]: Deploying daemon osd.0 on smithi094 2024-04-26T20:01:52.662 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:52 smithi094 ceph-mon[28050]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:52.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:52 smithi103 ceph-mon[32635]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:52.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:52 smithi204 ceph-mon[32618]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:54.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:54 smithi103 ceph-mon[32635]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:54.804 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:54 smithi094 ceph-mon[28050]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:54.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:54 smithi204 ceph-mon[32618]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:56.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:56 smithi094 ceph-mon[28050]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:56.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:56 smithi103 ceph-mon[32635]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:56.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:56 smithi204 ceph-mon[32618]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:58.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:58 smithi094 ceph-mon[28050]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:58.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:58.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:58.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:01:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:58.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:58 smithi204 ceph-mon[32618]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:58.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:58 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:58.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:58 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:58.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:01:58 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:01:59.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:58 smithi103 ceph-mon[32635]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:01:59.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:58 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:59.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:58 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:01:59.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:01:58 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:00.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:00 smithi204 ceph-mon[32618]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:02:00.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:00 smithi094 ceph-mon[28050]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:02:01.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:00 smithi103 ceph-mon[32635]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:02:02.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:02.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:02.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:02.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:02.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:02.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:02 smithi094 ceph-mon[28050]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:02:02.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:02 smithi094 ceph-mon[28050]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T20:02:02.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:02.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:02.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:02.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:02.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:02.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:02 smithi103 ceph-mon[32635]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:02:02.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:02 smithi103 ceph-mon[32635]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T20:02:02.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:02.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:02.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:02.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:02.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:02.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:02 smithi204 ceph-mon[32618]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:02:02.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:02 smithi204 ceph-mon[32618]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T20:02:03.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.292+0000 7f0220c00700 1 -- 172.21.15.94:0/1857357905 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0224063860 con 0x7f0208039890 2024-04-26T20:02:03.295 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 0 on host 'smithi094' 2024-04-26T20:02:03.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0208039890 msgr2=0x7f020803bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:03.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0208039890 0x7f020803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0214009680 tx=0x7f0214009260 comp rx=0 tx=0).stop 2024-04-26T20:02:03.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f022410b660 msgr2=0x7f02241a7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:03.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f022410b660 0x7f02241a7190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f021c00efb0 tx=0x7f021c00c3a0 comp rx=0 tx=0).stop 2024-04-26T20:02:03.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 shutdown_connections 2024-04-26T20:02:03.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0208039890 0x7f020803bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:03.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f022410b660 0x7f02241a7190 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:03.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0224109ac0 0x7f022419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:03.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 --2- 172.21.15.94:0/1857357905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022410acc0 0x7f02241a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:03.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.296+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 >> 172.21.15.94:0/1857357905 conn(0x7f0224078a90 msgr2=0x7f022410e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:02:03.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.297+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 shutdown_connections 2024-04-26T20:02:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:03.297+0000 7f022aa00700 1 -- 172.21.15.94:0/1857357905 wait complete. 2024-04-26T20:02:03.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:03 smithi094 ceph-mon[28050]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T20:02:03.490 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:03 smithi094 ceph-mon[28050]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T20:02:03.490 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:03 smithi094 ceph-mon[28050]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:02:03.490 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:03.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:03 smithi103 ceph-mon[32635]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T20:02:03.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:03 smithi103 ceph-mon[32635]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T20:02:03.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:03 smithi103 ceph-mon[32635]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:02:03.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:03.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:03 smithi204 ceph-mon[32618]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T20:02:03.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:03 smithi204 ceph-mon[32618]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T20:02:03.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:03 smithi204 ceph-mon[32618]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:02:03.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:04.084 DEBUG:teuthology.orchestra.run.smithi094:osd.0> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T20:02:04.088 INFO:tasks.cephadm:Deploying osd.1 on smithi103 with /dev/vg_nvme/lv_4... 2024-04-26T20:02:04.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T20:02:04.331 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:02:04.447 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:04 smithi103 ceph-mon[32635]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:02:04.447 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:04 smithi103 ceph-mon[32635]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T20:02:04.448 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:04.448 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:04.448 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:04.448 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:04.448 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:04.448 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:04 smithi103 ceph-mon[32635]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:02:04.448 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:04 smithi103 ceph-mon[32635]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' 2024-04-26T20:02:04.520 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:04 smithi094 ceph-mon[28050]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:02:04.520 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:04 smithi094 ceph-mon[28050]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T20:02:04.520 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:04.520 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:04.520 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:04.520 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:04.520 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:04.520 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:04 smithi094 ceph-mon[28050]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:02:04.521 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:04 smithi094 ceph-mon[28050]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' 2024-04-26T20:02:04.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:04 smithi204 ceph-mon[32618]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:02:04.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:04 smithi204 ceph-mon[32618]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T20:02:04.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:04.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:04.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:04.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:04.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:04.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:04 smithi204 ceph-mon[32618]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:02:04.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:04 smithi204 ceph-mon[32618]: from='osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901]' entity='osd.0' 2024-04-26T20:02:05.450 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:05 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:02:05.450 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:05 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:02:05.451 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:05.451 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:05 smithi103 ceph-mon[32635]: osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] boot 2024-04-26T20:02:05.451 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:05 smithi103 ceph-mon[32635]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T20:02:05.451 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:05.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:05 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:02:05.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:05 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:02:05.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:05.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:05 smithi094 ceph-mon[28050]: osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] boot 2024-04-26T20:02:05.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:05 smithi094 ceph-mon[28050]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T20:02:05.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:05.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:05 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:02:05.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:05 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:02:05.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:05.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:05 smithi204 ceph-mon[32618]: osd.0 [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] boot 2024-04-26T20:02:05.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:05 smithi204 ceph-mon[32618]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T20:02:05.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:02:05.992 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T20:02:05.992 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T20:02:05.992 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-26T20:02:05.992 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-26T20:02:05.993 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114576 s, 915 MB/s 2024-04-26T20:02:05.993 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-26T20:02:06.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:06 smithi094 ceph-mon[28050]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:06.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:06 smithi094 ceph-mon[28050]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T20:02:06.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:06 smithi103 ceph-mon[32635]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:06.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:06 smithi103 ceph-mon[32635]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T20:02:06.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_4 2024-04-26T20:02:06.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:06 smithi204 ceph-mon[32618]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:06.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:06 smithi204 ceph-mon[32618]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T20:02:06.839 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:02:08.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 -- 172.21.15.103:0/4234747158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb514105040 msgr2=0x7fb51410b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 --2- 172.21.15.103:0/4234747158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb514105040 0x7fb51410b8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00b0d0 tx=0x7fb50c00b3e0 comp rx=0 tx=0).stop 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 -- 172.21.15.103:0/4234747158 shutdown_connections 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 --2- 172.21.15.103:0/4234747158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb514105040 0x7fb51410b8f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 --2- 172.21.15.103:0/4234747158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5141034a0 0x7fb5141038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 --2- 172.21.15.103:0/4234747158 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5141046a0 0x7fb514104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 -- 172.21.15.103:0/4234747158 >> 172.21.15.103:0/4234747158 conn(0x7fb5140fea40 msgr2=0x7fb514100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 -- 172.21.15.103:0/4234747158 shutdown_connections 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 -- 172.21.15.103:0/4234747158 wait complete. 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 Processor -- start 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 -- start start 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5141034a0 0x7fb5141995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:08.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5141046a0 0x7fb514199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:08.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb514105040 0x7fb5141a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:08.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb51410efb0 con 0x7fb5141046a0 2024-04-26T20:02:08.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb51410ee30 con 0x7fb5141034a0 2024-04-26T20:02:08.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.174+0000 7fb51befa700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb51410f130 con 0x7fb514105040 2024-04-26T20:02:08.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb519c96700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5141034a0 0x7fb5141995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:08.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb519c96700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5141034a0 0x7fb5141995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47968/0 (socket says 172.21.15.103:47968) 2024-04-26T20:02:08.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb519c96700 1 -- 172.21.15.103:0/1434148462 learned_addr learned my addr 172.21.15.103:0/1434148462 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T20:02:08.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb51a497700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb514105040 0x7fb5141a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:08.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb519c96700 1 -- 172.21.15.103:0/1434148462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb514105040 msgr2=0x7fb5141a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb519c96700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb514105040 0x7fb5141a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb519c96700 1 -- 172.21.15.103:0/1434148462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5141046a0 msgr2=0x7fb514199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb519c96700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5141046a0 0x7fb514199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb519c96700 1 -- 172.21.15.103:0/1434148462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5141a1200 con 0x7fb5141034a0 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb519c96700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5141034a0 0x7fb5141995a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb51000c8f0 tx=0x7fb51000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb50affd700 1 -- 172.21.15.103:0/1434148462 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510004220 con 0x7fb5141034a0 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb50affd700 1 -- 172.21.15.103:0/1434148462 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb510007db0 con 0x7fb5141034a0 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5141a14f0 con 0x7fb5141034a0 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb50affd700 1 -- 172.21.15.103:0/1434148462 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510003d00 con 0x7fb5141034a0 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.175+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5141a1a60 con 0x7fb5141034a0 2024-04-26T20:02:08.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.176+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb51406a110 con 0x7fb5141034a0 2024-04-26T20:02:08.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.178+0000 7fb50affd700 1 -- 172.21.15.103:0/1434148462 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fb510007900 con 0x7fb5141034a0 2024-04-26T20:02:08.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.178+0000 7fb50affd700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5000a3bb0 0x7fb5000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:08.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.178+0000 7fb50affd700 1 -- 172.21.15.103:0/1434148462 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fb5100ce990 con 0x7fb5141034a0 2024-04-26T20:02:08.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.179+0000 7fb519495700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5000a3bb0 0x7fb5000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:08.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.180+0000 7fb519495700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5000a3bb0 0x7fb5000a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb5040097b0 tx=0x7fb504006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:02:08.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.182+0000 7fb50affd700 1 -- 172.21.15.103:0/1434148462 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb510014030 con 0x7fb5141034a0 2024-04-26T20:02:08.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:08.387+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb514063860 con 0x7fb5000a3bb0 2024-04-26T20:02:08.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:08 smithi094 ceph-mon[28050]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:08.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:02:08.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:02:08.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:08.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:08 smithi103 ceph-mon[32635]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:08.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:02:08.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:02:08.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:08.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:08 smithi204 ceph-mon[32618]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:08.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:02:08.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:02:08.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:09.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:09 smithi103 ceph-mon[32635]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:02:09.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:02:09.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:09.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:09.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:09 smithi204 ceph-mon[32618]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:02:09.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:02:09.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:09.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:09.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:09 smithi094 ceph-mon[28050]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:02:09.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:02:09.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:09.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:09.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:09.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:10 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi094 2024-04-26T20:02:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:10 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi094 to 17089M 2024-04-26T20:02:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:10 smithi103 ceph-mon[32635]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:10.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:10 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi094 2024-04-26T20:02:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:10 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi094 to 17089M 2024-04-26T20:02:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:10 smithi204 ceph-mon[32618]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:10 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi094 2024-04-26T20:02:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:10 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi094 to 17089M 2024-04-26T20:02:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:10 smithi094 ceph-mon[28050]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:11.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:11 smithi103 ceph-mon[32635]: from='client.? 172.21.15.103:0/1710491322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6efa3-ce90-4d2d-8fb8-4a2665258434"}]: dispatch 2024-04-26T20:02:11.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:11 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6efa3-ce90-4d2d-8fb8-4a2665258434"}]: dispatch 2024-04-26T20:02:11.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:11 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eef6efa3-ce90-4d2d-8fb8-4a2665258434"}]': finished 2024-04-26T20:02:11.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:11 smithi103 ceph-mon[32635]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T20:02:11.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:11.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:11 smithi103 ceph-mon[32635]: from='client.? 172.21.15.103:0/1515825093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:02:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:11 smithi204 ceph-mon[32618]: from='client.? 172.21.15.103:0/1710491322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6efa3-ce90-4d2d-8fb8-4a2665258434"}]: dispatch 2024-04-26T20:02:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:11 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6efa3-ce90-4d2d-8fb8-4a2665258434"}]: dispatch 2024-04-26T20:02:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:11 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eef6efa3-ce90-4d2d-8fb8-4a2665258434"}]': finished 2024-04-26T20:02:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:11 smithi204 ceph-mon[32618]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T20:02:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:11 smithi204 ceph-mon[32618]: from='client.? 172.21.15.103:0/1515825093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:02:11.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:11 smithi094 ceph-mon[28050]: from='client.? 172.21.15.103:0/1710491322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6efa3-ce90-4d2d-8fb8-4a2665258434"}]: dispatch 2024-04-26T20:02:11.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:11 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eef6efa3-ce90-4d2d-8fb8-4a2665258434"}]: dispatch 2024-04-26T20:02:11.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:11 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eef6efa3-ce90-4d2d-8fb8-4a2665258434"}]': finished 2024-04-26T20:02:11.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:11 smithi094 ceph-mon[28050]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T20:02:11.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:11.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:11 smithi094 ceph-mon[28050]: from='client.? 172.21.15.103:0/1515825093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:02:12.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:12 smithi103 ceph-mon[32635]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:12.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:12 smithi204 ceph-mon[32618]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:12 smithi094 ceph-mon[28050]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:14.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:14 smithi103 ceph-mon[32635]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:14.789 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:14 smithi094 ceph-mon[28050]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:14.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:14 smithi204 ceph-mon[32618]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:16.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:16 smithi094 ceph-mon[28050]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:16.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:16 smithi103 ceph-mon[32635]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:16.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:16 smithi204 ceph-mon[32618]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:18.638 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:18 smithi103 ceph-mon[32635]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:18.638 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:18 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T20:02:18.638 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:18 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:18.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:18 smithi204 ceph-mon[32618]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:18.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:18 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T20:02:18.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:18 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:18 smithi094 ceph-mon[28050]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:18 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T20:02:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:18 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:19.679 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:19 smithi103 ceph-mon[32635]: Deploying daemon osd.1 on smithi103 2024-04-26T20:02:19.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:19 smithi204 ceph-mon[32618]: Deploying daemon osd.1 on smithi103 2024-04-26T20:02:19.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:19 smithi094 ceph-mon[28050]: Deploying daemon osd.1 on smithi103 2024-04-26T20:02:20.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:20 smithi204 ceph-mon[32618]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:20.931 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:20 smithi103 ceph-mon[32635]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:20.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:20 smithi094 ceph-mon[28050]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:22.689 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:22 smithi103 ceph-mon[32635]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:22.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:22 smithi204 ceph-mon[32618]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:22.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:22 smithi094 ceph-mon[28050]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:24.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:24.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:24.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:24.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:23 smithi094 ceph-mon[28050]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:23 smithi103 ceph-mon[32635]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:24.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:24.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:24.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:24.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:23 smithi204 ceph-mon[32618]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:26.426 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 1 on host 'smithi103' 2024-04-26T20:02:26.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.424+0000 7fb50affd700 1 -- 172.21.15.103:0/1434148462 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb514063860 con 0x7fb5000a3bb0 2024-04-26T20:02:26.426 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:26.427 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:26.427 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:26.427 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:26.427 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:26.427 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:26 smithi103 ceph-mon[32635]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:26.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5000a3bb0 msgr2=0x7fb5000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:26.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5000a3bb0 0x7fb5000a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb5040097b0 tx=0x7fb504006d20 comp rx=0 tx=0).stop 2024-04-26T20:02:26.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5141034a0 msgr2=0x7fb5141995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:26.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5141034a0 0x7fb5141995a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb51000c8f0 tx=0x7fb51000ccb0 comp rx=0 tx=0).stop 2024-04-26T20:02:26.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 shutdown_connections 2024-04-26T20:02:26.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb514105040 0x7fb5141a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:26.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5000a3bb0 0x7fb5000a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:26.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5141034a0 0x7fb5141995a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:26.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 --2- 172.21.15.103:0/1434148462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5141046a0 0x7fb514199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:26.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 >> 172.21.15.103:0/1434148462 conn(0x7fb5140fea40 msgr2=0x7fb514107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:02:26.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 shutdown_connections 2024-04-26T20:02:26.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:02:26.427+0000 7fb51befa700 1 -- 172.21.15.103:0/1434148462 wait complete. 2024-04-26T20:02:26.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:26.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:26.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:26.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:26.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:26.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:26 smithi204 ceph-mon[32618]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:26.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:26.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:26.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:26.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:26.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:26 smithi094 ceph-mon[28050]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:26.980 DEBUG:teuthology.orchestra.run.smithi103:osd.1> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T20:02:26.982 INFO:tasks.cephadm:Deploying osd.2 on smithi204 with /dev/vg_nvme/lv_4... 2024-04-26T20:02:26.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T20:02:27.227 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:02:27.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:27.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:27.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:27.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:27 smithi103 ceph-mon[32635]: from='osd.1 [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:02:27.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:27 smithi103 ceph-mon[32635]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:02:27.619 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:27.620 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:27.620 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:27.620 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:27 smithi204 ceph-mon[32618]: from='osd.1 [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:02:27.620 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:27 smithi204 ceph-mon[32618]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:02:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:27.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:27.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:27 smithi094 ceph-mon[28050]: from='osd.1 [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:02:27.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:27 smithi094 ceph-mon[28050]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:02:28.542 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:28 smithi204 ceph-mon[32618]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T20:02:28.542 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:28 smithi204 ceph-mon[32618]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T20:02:28.542 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:28 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:28.543 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:28 smithi204 ceph-mon[32618]: from='osd.1 [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:02:28.543 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:28 smithi204 ceph-mon[32618]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:02:28.543 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:28 smithi204 ceph-mon[32618]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:28.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:28 smithi094 ceph-mon[28050]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T20:02:28.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:28 smithi094 ceph-mon[28050]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T20:02:28.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:28 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:28.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:28 smithi094 ceph-mon[28050]: from='osd.1 [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:02:28.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:28 smithi094 ceph-mon[28050]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:02:28.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:28 smithi094 ceph-mon[28050]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:28.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:28 smithi103 ceph-mon[32635]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T20:02:28.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:28 smithi103 ceph-mon[32635]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T20:02:28.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:28 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:28.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:28 smithi103 ceph-mon[32635]: from='osd.1 [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:02:28.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:28 smithi103 ceph-mon[32635]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:02:28.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:28 smithi103 ceph-mon[32635]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:29.099 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T20:02:29.099 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T20:02:29.099 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-04-26T20:02:29.099 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-04-26T20:02:29.099 INFO:teuthology.orchestra.run.smithi204.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121601 s, 862 MB/s 2024-04-26T20:02:29.100 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 2024-04-26T20:02:29.100 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-04-26T20:02:29.440 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:29 smithi103 ceph-mon[32635]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:02:29.440 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:29 smithi103 ceph-mon[32635]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T20:02:29.440 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:29.441 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:29 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1[37217]: 2024-04-26T20:02:29.151+0000 7fd9d5f45700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T20:02:29.441 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:29 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1[37217]: 2024-04-26T20:02:29.155+0000 7fd9cd52f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:02:29.655 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:29 smithi204 ceph-mon[32618]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:02:29.655 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:29 smithi204 ceph-mon[32618]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T20:02:29.655 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:29.655 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:29.655 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:29 smithi204 ceph-mon[32618]: from='osd.1 ' entity='osd.1' 2024-04-26T20:02:29.655 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:29 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:29.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_4 2024-04-26T20:02:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:29 smithi094 ceph-mon[28050]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:02:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:29 smithi094 ceph-mon[28050]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T20:02:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:29 smithi094 ceph-mon[28050]: from='osd.1 ' entity='osd.1' 2024-04-26T20:02:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:29 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:29.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:29.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:29 smithi103 ceph-mon[32635]: from='osd.1 ' entity='osd.1' 2024-04-26T20:02:29.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:29 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:29.948 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: osd.1 [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] boot 2024-04-26T20:02:30.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T20:02:30.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:30.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:02:30.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: osd.1 [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] boot 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T20:02:30.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:30.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:02:30.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:02:30.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:02:30.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:02:30.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:30.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:30.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:30.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: osd.1 [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] boot 2024-04-26T20:02:30.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T20:02:30.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:02:31.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.350+0000 7f451a86c700 1 -- 172.21.15.204:0/1310294977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514104680 msgr2=0x7f4514104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.350+0000 7f451a86c700 1 --2- 172.21.15.204:0/1310294977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514104680 0x7f4514104ae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4508009990 tx=0x7f4508009ca0 comp rx=0 tx=0).stop 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.350+0000 7f451a86c700 1 -- 172.21.15.204:0/1310294977 shutdown_connections 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.350+0000 7f451a86c700 1 --2- 172.21.15.204:0/1310294977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4514105020 0x7f451410b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.350+0000 7f451a86c700 1 --2- 172.21.15.204:0/1310294977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4514103480 0x7f45141038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.350+0000 7f451a86c700 1 --2- 172.21.15.204:0/1310294977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514104680 0x7f4514104ae0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.350+0000 7f451a86c700 1 -- 172.21.15.204:0/1310294977 >> 172.21.15.204:0/1310294977 conn(0x7f45140fea20 msgr2=0x7f4514100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.350+0000 7f451a86c700 1 -- 172.21.15.204:0/1310294977 shutdown_connections 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.350+0000 7f451a86c700 1 -- 172.21.15.204:0/1310294977 wait complete. 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f451a86c700 1 Processor -- start 2024-04-26T20:02:31.352 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f451a86c700 1 -- start start 2024-04-26T20:02:31.354 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f451a86c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4514103480 0x7f45140771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:31.354 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f451a86c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4514104680 0x7f4514075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:31.354 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f451a86c700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514105020 0x7f4514075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:31.354 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f451a86c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f451410ef70 con 0x7f4514103480 2024-04-26T20:02:31.354 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f451a86c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f451410edf0 con 0x7f4514104680 2024-04-26T20:02:31.354 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f451a86c700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f451410f0f0 con 0x7f4514105020 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f4518e09700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514105020 0x7f4514075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f4518e09700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514105020 0x7f4514075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:37626/0 (socket says 172.21.15.204:37626) 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f4518e09700 1 -- 172.21.15.204:0/1574076999 learned_addr learned my addr 172.21.15.204:0/1574076999 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f45137fe700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4514104680 0x7f4514075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f4518e09700 1 -- 172.21.15.204:0/1574076999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4514104680 msgr2=0x7f4514075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f4518e09700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4514104680 0x7f4514075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f4518e09700 1 -- 172.21.15.204:0/1574076999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4514103480 msgr2=0x7f45140771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f4518e09700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4514103480 0x7f45140771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.351+0000 7f4518e09700 1 -- 172.21.15.204:0/1574076999 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4514076600 con 0x7f4514105020 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.352+0000 7f4518e09700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514105020 0x7f4514075d50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f450c00b970 tx=0x7f450c00bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.352+0000 7f45117fa700 1 -- 172.21.15.204:0/1574076999 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c019070 con 0x7f4514105020 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.352+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45141a37a0 con 0x7f4514105020 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.352+0000 7f45117fa700 1 -- 172.21.15.204:0/1574076999 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f450c00cde0 con 0x7f4514105020 2024-04-26T20:02:31.355 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.352+0000 7f45117fa700 1 -- 172.21.15.204:0/1574076999 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c007370 con 0x7f4514105020 2024-04-26T20:02:31.356 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.352+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45141a3c60 con 0x7f4514105020 2024-04-26T20:02:31.356 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.353+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f451406a110 con 0x7f4514105020 2024-04-26T20:02:31.356 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.354+0000 7f45117fa700 1 -- 172.21.15.204:0/1574076999 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f450c01c070 con 0x7f4514105020 2024-04-26T20:02:31.356 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.354+0000 7f45117fa700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f44fc0a3bb0 0x7f44fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:31.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.354+0000 7f45117fa700 1 -- 172.21.15.204:0/1574076999 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f450c0ce7a0 con 0x7f4514105020 2024-04-26T20:02:31.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.355+0000 7f4513fff700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f44fc0a3bb0 0x7f44fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:31.357 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.356+0000 7f4513fff700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f44fc0a3bb0 0x7f44fc0a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f45141044e0 tx=0x7f4504005ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:02:31.359 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.358+0000 7f45117fa700 1 -- 172.21.15.204:0/1574076999 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f450c023070 con 0x7f4514105020 2024-04-26T20:02:31.567 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:31.566+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4514063860 con 0x7f44fc0a3bb0 2024-04-26T20:02:31.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:31 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi103 2024-04-26T20:02:31.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:31 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi103 to 17088M 2024-04-26T20:02:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:31 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi103 2024-04-26T20:02:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:31 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi103 to 17088M 2024-04-26T20:02:31.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:31 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi103 2024-04-26T20:02:31.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:31 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi103 to 17088M 2024-04-26T20:02:32.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:32 smithi094 ceph-mon[28050]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:32.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:32 smithi094 ceph-mon[28050]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T20:02:32.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:32 smithi094 ceph-mon[28050]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:02:32.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:02:32.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:02:32.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:32.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:32 smithi103 ceph-mon[32635]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:32.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:32 smithi103 ceph-mon[32635]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T20:02:32.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:32 smithi103 ceph-mon[32635]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:02:32.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:02:32.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:02:32.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:32.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:32 smithi204 ceph-mon[32618]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:32 smithi204 ceph-mon[32618]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T20:02:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:32 smithi204 ceph-mon[32618]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:02:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:02:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:02:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:34.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:34 smithi204 ceph-mon[32618]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:34.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:34 smithi204 ceph-mon[32618]: from='client.? 172.21.15.204:0/3170411409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "939eb12b-359a-4c5e-8e2f-c94e6512050e"}]: dispatch 2024-04-26T20:02:34.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:34 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "939eb12b-359a-4c5e-8e2f-c94e6512050e"}]: dispatch 2024-04-26T20:02:34.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:34 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "939eb12b-359a-4c5e-8e2f-c94e6512050e"}]': finished 2024-04-26T20:02:34.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:34 smithi204 ceph-mon[32618]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T20:02:34.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:34 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:34.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:34 smithi094 ceph-mon[28050]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:34.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:34 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/3170411409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "939eb12b-359a-4c5e-8e2f-c94e6512050e"}]: dispatch 2024-04-26T20:02:34.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:34 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "939eb12b-359a-4c5e-8e2f-c94e6512050e"}]: dispatch 2024-04-26T20:02:34.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:34 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "939eb12b-359a-4c5e-8e2f-c94e6512050e"}]': finished 2024-04-26T20:02:34.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:34 smithi094 ceph-mon[28050]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T20:02:34.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:34 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:34.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:34 smithi103 ceph-mon[32635]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:34.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:34 smithi103 ceph-mon[32635]: from='client.? 172.21.15.204:0/3170411409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "939eb12b-359a-4c5e-8e2f-c94e6512050e"}]: dispatch 2024-04-26T20:02:34.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:34 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "939eb12b-359a-4c5e-8e2f-c94e6512050e"}]: dispatch 2024-04-26T20:02:34.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:34 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "939eb12b-359a-4c5e-8e2f-c94e6512050e"}]': finished 2024-04-26T20:02:34.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:34 smithi103 ceph-mon[32635]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T20:02:34.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:34 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:35.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:35 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/3460501162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:02:35.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:35 smithi103 ceph-mon[32635]: from='client.? 172.21.15.204:0/3460501162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:02:35.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:35 smithi204 ceph-mon[32618]: from='client.? 172.21.15.204:0/3460501162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:02:36.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:36 smithi094 ceph-mon[28050]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:36.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:36 smithi103 ceph-mon[32635]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:36.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:36 smithi204 ceph-mon[32618]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:38.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:38 smithi204 ceph-mon[32618]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:38.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:38 smithi094 ceph-mon[28050]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:38.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:38 smithi103 ceph-mon[32635]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:40.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:40 smithi103 ceph-mon[32635]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:40.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:40 smithi204 ceph-mon[32618]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:40.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:40 smithi094 ceph-mon[28050]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:41 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T20:02:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:41 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:41.833 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:41 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T20:02:41.833 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:41 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:41.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:41 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T20:02:41.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:41 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:42 smithi204 ceph-mon[32618]: Deploying daemon osd.2 on smithi204 2024-04-26T20:02:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:42 smithi204 ceph-mon[32618]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:42.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:42 smithi103 ceph-mon[32635]: Deploying daemon osd.2 on smithi204 2024-04-26T20:02:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:42 smithi103 ceph-mon[32635]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:42.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:42 smithi094 ceph-mon[28050]: Deploying daemon osd.2 on smithi204 2024-04-26T20:02:42.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:42 smithi094 ceph-mon[28050]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:44.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:44 smithi103 ceph-mon[32635]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:44.790 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:44 smithi094 ceph-mon[28050]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:44 smithi204 ceph-mon[32618]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:46.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:46 smithi094 ceph-mon[28050]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:46.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:46 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:46.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:46 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:46.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:46 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:46.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:46 smithi103 ceph-mon[32635]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:46.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:46 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:46.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:46 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:46.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:46 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:46.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:46 smithi204 ceph-mon[32618]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:46.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:46 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:46.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:46 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:46.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:46 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:48.701 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:48 smithi204 ceph-mon[32618]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:48.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:48 smithi103 ceph-mon[32635]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:48.986 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:48 smithi094 ceph-mon[28050]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:49.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.754+0000 7f45117fa700 1 -- 172.21.15.204:0/1574076999 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4514063860 con 0x7f44fc0a3bb0 2024-04-26T20:02:49.756 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 2 on host 'smithi204' 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.756+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f44fc0a3bb0 msgr2=0x7f44fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.756+0000 7f451a86c700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f44fc0a3bb0 0x7f44fc0a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f45141044e0 tx=0x7f4504005ca0 comp rx=0 tx=0).stop 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.756+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514105020 msgr2=0x7f4514075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.756+0000 7f451a86c700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514105020 0x7f4514075d50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f450c00b970 tx=0x7f450c00bc80 comp rx=0 tx=0).stop 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.756+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 shutdown_connections 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.757+0000 7f451a86c700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4514105020 0x7f4514075d50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.757+0000 7f451a86c700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4514103480 0x7f45140771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.757+0000 7f451a86c700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f44fc0a3bb0 0x7f44fc0a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.757+0000 7f451a86c700 1 --2- 172.21.15.204:0/1574076999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4514104680 0x7f4514075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.757+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 >> 172.21.15.204:0/1574076999 conn(0x7f45140fea20 msgr2=0x7f45141079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:02:49.758 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.757+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 shutdown_connections 2024-04-26T20:02:49.759 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:02:49.757+0000 7f451a86c700 1 -- 172.21.15.204:0/1574076999 wait complete. 2024-04-26T20:02:50.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.103 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:49 smithi204 ceph-mon[32618]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:49 smithi094 ceph-mon[28050]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:50.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:50.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:50.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:49 smithi103 ceph-mon[32635]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:50.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:50.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:02:50.444 DEBUG:teuthology.orchestra.run.smithi204:osd.2> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T20:02:50.447 INFO:tasks.cephadm:Deploying osd.3 on smithi094 with /dev/vg_nvme/lv_3... 2024-04-26T20:02:50.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T20:02:50.887 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:02:51.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:50 smithi204 ceph-mon[32618]: from='osd.2 [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:02:51.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:50 smithi204 ceph-mon[32618]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:02:51.201 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:50 smithi094 ceph-mon[28050]: from='osd.2 [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:02:51.201 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:50 smithi094 ceph-mon[28050]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:02:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:50 smithi103 ceph-mon[32635]: from='osd.2 [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:02:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:50 smithi103 ceph-mon[32635]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:02:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:51 smithi204 ceph-mon[32618]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T20:02:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:51 smithi204 ceph-mon[32618]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T20:02:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:51 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:51 smithi204 ceph-mon[32618]: from='osd.2 [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:02:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:51 smithi204 ceph-mon[32618]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:02:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:51 smithi204 ceph-mon[32618]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:52.132 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:51 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2[37758]: 2024-04-26T20:02:51.884+0000 7f9fca787700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T20:02:52.132 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:51 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2[37758]: 2024-04-26T20:02:51.888+0000 7f9fc657a700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:02:52.223 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:51 smithi094 ceph-mon[28050]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T20:02:52.223 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:51 smithi094 ceph-mon[28050]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T20:02:52.223 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:51 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:52.223 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:51 smithi094 ceph-mon[28050]: from='osd.2 [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:02:52.224 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:51 smithi094 ceph-mon[28050]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:02:52.224 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:51 smithi094 ceph-mon[28050]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:51 smithi103 ceph-mon[32635]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T20:02:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:51 smithi103 ceph-mon[32635]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T20:02:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:51 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:51 smithi103 ceph-mon[32635]: from='osd.2 [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:02:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:51 smithi103 ceph-mon[32635]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:02:52.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:51 smithi103 ceph-mon[32635]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:53.210 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:52 smithi204 ceph-mon[32618]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:02:53.210 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:52 smithi204 ceph-mon[32618]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T20:02:53.210 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:53.210 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:52 smithi094 ceph-mon[28050]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:02:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:52 smithi094 ceph-mon[28050]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T20:02:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:52 smithi103 ceph-mon[32635]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:02:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:52 smithi103 ceph-mon[32635]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T20:02:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:53.670 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T20:02:53.670 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T20:02:53.670 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-04-26T20:02:53.670 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-04-26T20:02:53.670 INFO:teuthology.orchestra.run.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0342926 s, 306 MB/s 2024-04-26T20:02:53.671 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-04-26T20:02:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:02:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:02:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: osd.2 [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] boot 2024-04-26T20:02:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T20:02:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi204 2024-04-26T20:02:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:53.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:02:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi204 to 17088M 2024-04-26T20:02:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: pgmap v89: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:53.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:53 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:02:54.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:02:54.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: osd.2 [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] boot 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi204 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi204 to 17088M 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: pgmap v89: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:53 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:02:54.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:02:54.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:02:54.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: osd.2 [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] boot 2024-04-26T20:02:54.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T20:02:54.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:02:54.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi204 2024-04-26T20:02:54.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:54.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:54.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:02:54.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi204 to 17088M 2024-04-26T20:02:54.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:54.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:54.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:02:54.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:54.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: pgmap v89: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:02:54.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:53 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:02:54.541 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_3 2024-04-26T20:02:54.985 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:02:55.184 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T20:02:55.184 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:54 smithi103 ceph-mon[32635]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T20:02:55.184 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:02:55.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T20:02:55.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:54 smithi094 ceph-mon[28050]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T20:02:55.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:02:55.238 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44215]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:02:55.238 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44215]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.238 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44215]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.238 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44215]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.238 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44218]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:02:55.239 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44218]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.239 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44218]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.239 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44218]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.270 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T20:02:55.270 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:54 smithi204 ceph-mon[32618]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T20:02:55.270 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:02:55.476 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40915]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:02:55.476 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40915]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.476 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40915]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.476 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40915]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.476 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40918]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:02:55.476 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40918]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.476 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40918]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.476 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40918]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.621 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41414]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:02:55.621 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41414]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.621 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41414]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.621 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41414]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.621 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41417]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:02:55.621 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41417]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.621 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41417]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.622 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41417]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.622 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41408]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:02:55.622 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41408]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.622 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41408]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.622 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41408]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.622 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41411]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:02:55.622 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41411]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.622 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41411]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.622 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:02:55 smithi204 sudo[41411]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.635 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44224]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:02:55.635 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44224]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.635 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44224]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.635 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44224]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.635 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44228]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:02:55.635 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44228]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.635 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44228]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.635 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 sudo[44228]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40921]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:02:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40921]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40921]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40921]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:55.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40924]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:02:55.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40924]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:02:55.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40924]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:02:55.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 sudo[40924]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:02:56.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:02:56.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T20:02:56.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:56.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:56.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:56.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:02:56.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:02:56.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:02:56.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:55 smithi094 ceph-mon[28050]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:02:56.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:02:56.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:02:56.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:02:56.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:02:56.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:02:56.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:02:56.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:02:56.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:02:56.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:02:56.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:55 smithi103 ceph-mon[32635]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:02:56.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:02:56.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T20:02:56.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:56.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:56.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:02:56.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:02:56.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:02:56.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:02:56.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:02:56.384 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:55 smithi204 ceph-mon[32618]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:02:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.215+0000 7f73c3a00700 1 -- 172.21.15.94:0/1950674717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc105c70 msgr2=0x7f73bc10db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.215+0000 7f73c3a00700 1 --2- 172.21.15.94:0/1950674717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc105c70 0x7f73bc10db00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f73b400b0a0 tx=0x7f73b400b3b0 comp rx=0 tx=0).stop 2024-04-26T20:02:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.215+0000 7f73c3a00700 1 -- 172.21.15.94:0/1950674717 shutdown_connections 2024-04-26T20:02:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.215+0000 7f73c3a00700 1 --2- 172.21.15.94:0/1950674717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc105c70 0x7f73bc10db00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.215+0000 7f73c3a00700 1 --2- 172.21.15.94:0/1950674717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73bc106d90 0x7f73bc1071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.215+0000 7f73c3a00700 1 --2- 172.21.15.94:0/1950674717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73bc1052d0 0x7f73bc105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:57.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.215+0000 7f73c3a00700 1 -- 172.21.15.94:0/1950674717 >> 172.21.15.94:0/1950674717 conn(0x7f73bc100c70 msgr2=0x7f73bc103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:02:57.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:56 smithi094 ceph-mon[28050]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T20:02:57.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:56 smithi094 ceph-mon[28050]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T20:02:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.215+0000 7f73c3a00700 1 -- 172.21.15.94:0/1950674717 shutdown_connections 2024-04-26T20:02:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.215+0000 7f73c3a00700 1 -- 172.21.15.94:0/1950674717 wait complete. 2024-04-26T20:02:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.216+0000 7f73c3a00700 1 Processor -- start 2024-04-26T20:02:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73c3a00700 1 -- start start 2024-04-26T20:02:57.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73c3a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73bc1052d0 0x7f73bc19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73c3a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc105c70 0x7f73bc19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73c3a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73bc106d90 0x7f73bc1a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73c3a00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc111230 con 0x7f73bc105c70 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73c3a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc1110b0 con 0x7f73bc106d90 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73c3a00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc1113b0 con 0x7f73bc1052d0 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73bbe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc105c70 0x7f73bc19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73c1000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73bc1052d0 0x7f73bc19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73bbe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc105c70 0x7f73bc19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54388/0 (socket says 172.21.15.94:54388) 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.217+0000 7f73bbe00700 1 -- 172.21.15.94:0/3275513041 learned_addr learned my addr 172.21.15.94:0/3275513041 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.218+0000 7f73c1a00700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73bc106d90 0x7f73bc1a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.218+0000 7f73c1000700 1 -- 172.21.15.94:0/3275513041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73bc106d90 msgr2=0x7f73bc1a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.218+0000 7f73c1000700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73bc106d90 0x7f73bc1a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.218+0000 7f73c1000700 1 -- 172.21.15.94:0/3275513041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc105c70 msgr2=0x7f73bc19bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:02:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.218+0000 7f73c1000700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc105c70 0x7f73bc19bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:02:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.218+0000 7f73c1000700 1 -- 172.21.15.94:0/3275513041 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc1a3450 con 0x7f73bc1052d0 2024-04-26T20:02:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.218+0000 7f73c1000700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73bc1052d0 0x7f73bc19b7f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f73b000e9f0 tx=0x7f73b000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:02:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.218+0000 7f73b9600700 1 -- 172.21.15.94:0/3275513041 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b000cb70 con 0x7f73bc1052d0 2024-04-26T20:02:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.219+0000 7f73b9600700 1 -- 172.21.15.94:0/3275513041 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73b000ee90 con 0x7f73bc1052d0 2024-04-26T20:02:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.219+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73bc1a3740 con 0x7f73bc1052d0 2024-04-26T20:02:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.219+0000 7f73b9600700 1 -- 172.21.15.94:0/3275513041 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b0018680 con 0x7f73bc1052d0 2024-04-26T20:02:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.219+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73bc1a3cd0 con 0x7f73bc1052d0 2024-04-26T20:02:57.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.220+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73bc06a110 con 0x7f73bc1052d0 2024-04-26T20:02:57.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.222+0000 7f73b9600700 1 -- 172.21.15.94:0/3275513041 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f73b001f020 con 0x7f73bc1052d0 2024-04-26T20:02:57.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.223+0000 7f73b9600700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f73a40a3cc0 0x7f73a40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:02:57.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.223+0000 7f73b9600700 1 -- 172.21.15.94:0/3275513041 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f73b00cefa0 con 0x7f73bc1052d0 2024-04-26T20:02:57.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.223+0000 7f73bbe00700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f73a40a3cc0 0x7f73a40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:02:57.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.224+0000 7f73bbe00700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f73a40a3cc0 0x7f73a40a6180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009680 tx=0x7f73a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:02:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.226+0000 7f73b9600700 1 -- 172.21.15.94:0/3275513041 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f73b009ccb0 con 0x7f73bc1052d0 2024-04-26T20:02:57.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:56 smithi103 ceph-mon[32635]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T20:02:57.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:56 smithi103 ceph-mon[32635]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T20:02:57.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:56 smithi204 ceph-mon[32618]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T20:02:57.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:56 smithi204 ceph-mon[32618]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T20:02:57.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:02:57.467+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f73bc063860 con 0x7f73a40a3cc0 2024-04-26T20:02:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:57 smithi094 ceph-mon[28050]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:02:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:57 smithi094 ceph-mon[28050]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:02:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:02:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:02:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:02:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:57 smithi103 ceph-mon[32635]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:02:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:57 smithi103 ceph-mon[32635]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:02:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:02:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:02:58.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:02:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:02:58.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:57 smithi204 ceph-mon[32618]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:02:58.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:57 smithi204 ceph-mon[32618]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:02:58.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:02:58.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:02:58.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:02:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:00.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:00 smithi103 ceph-mon[32635]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:00.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:00 smithi204 ceph-mon[32618]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:00.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:00 smithi094 ceph-mon[28050]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:01.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:01 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4216818245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa4d0799-6fb8-439e-a987-91a977ac9172"}]: dispatch 2024-04-26T20:03:01.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:01 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa4d0799-6fb8-439e-a987-91a977ac9172"}]: dispatch 2024-04-26T20:03:01.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:01 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa4d0799-6fb8-439e-a987-91a977ac9172"}]': finished 2024-04-26T20:03:01.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:01 smithi103 ceph-mon[32635]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T20:03:01.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:01 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:01.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:01 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4216818245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa4d0799-6fb8-439e-a987-91a977ac9172"}]: dispatch 2024-04-26T20:03:01.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:01 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa4d0799-6fb8-439e-a987-91a977ac9172"}]: dispatch 2024-04-26T20:03:01.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:01 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa4d0799-6fb8-439e-a987-91a977ac9172"}]': finished 2024-04-26T20:03:01.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:01 smithi204 ceph-mon[32618]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T20:03:01.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:01 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:01.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:01 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4216818245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa4d0799-6fb8-439e-a987-91a977ac9172"}]: dispatch 2024-04-26T20:03:01.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:01 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa4d0799-6fb8-439e-a987-91a977ac9172"}]: dispatch 2024-04-26T20:03:01.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:01 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa4d0799-6fb8-439e-a987-91a977ac9172"}]': finished 2024-04-26T20:03:01.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:01 smithi094 ceph-mon[28050]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T20:03:01.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:02.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:02 smithi103 ceph-mon[32635]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-04-26T20:03:02.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:02 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1434795680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:03:02.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:02 smithi204 ceph-mon[32618]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-04-26T20:03:02.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:02 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1434795680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:03:02.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:02 smithi094 ceph-mon[28050]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-04-26T20:03:02.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:02 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1434795680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:03:04.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:04 smithi103 ceph-mon[32635]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-04-26T20:03:04.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:04 smithi204 ceph-mon[32618]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-04-26T20:03:04.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:04 smithi094 ceph-mon[28050]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-04-26T20:03:06.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:06 smithi094 ceph-mon[28050]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 968 B/s rd, 28 KiB/s wr, 2 op/s 2024-04-26T20:03:06.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:06 smithi103 ceph-mon[32635]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 968 B/s rd, 28 KiB/s wr, 2 op/s 2024-04-26T20:03:06.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:06 smithi204 ceph-mon[32618]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 968 B/s rd, 28 KiB/s wr, 2 op/s 2024-04-26T20:03:08.593 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:08 smithi094 ceph-mon[28050]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-26T20:03:08.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:08 smithi103 ceph-mon[32635]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-26T20:03:08.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:08 smithi204 ceph-mon[32618]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-26T20:03:10.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:10 smithi204 ceph-mon[32618]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-26T20:03:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:10 smithi094 ceph-mon[28050]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-26T20:03:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:10 smithi103 ceph-mon[32635]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-26T20:03:11.881 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T20:03:11.881 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:11.881 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:11 smithi094 ceph-mon[28050]: Deploying daemon osd.3 on smithi094 2024-04-26T20:03:11.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T20:03:11.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:11 smithi204 ceph-mon[32618]: Deploying daemon osd.3 on smithi094 2024-04-26T20:03:11.935 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T20:03:11.935 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:11.936 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:11 smithi103 ceph-mon[32635]: Deploying daemon osd.3 on smithi094 2024-04-26T20:03:12.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:12 smithi094 ceph-mon[28050]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:12.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:12 smithi204 ceph-mon[32618]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:13.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:12 smithi103 ceph-mon[32635]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:14.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:14 smithi204 ceph-mon[32618]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:14.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:14 smithi094 ceph-mon[28050]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:15.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:14 smithi103 ceph-mon[32635]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:16.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:16 smithi094 ceph-mon[28050]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:16.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:16 smithi204 ceph-mon[32618]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:17.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:16 smithi103 ceph-mon[32635]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:18.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:18 smithi094 ceph-mon[28050]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:18.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:18 smithi204 ceph-mon[32618]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:19.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:18 smithi103 ceph-mon[32635]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:20.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:20.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:20.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:20.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:20 smithi094 ceph-mon[28050]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:20.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:20.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:20.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:20.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:20 smithi103 ceph-mon[32635]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:20.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:20.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:20.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:20.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:20 smithi204 ceph-mon[32618]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:22.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:22 smithi094 ceph-mon[28050]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:22.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:22.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:22.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:22.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:22.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:22.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:22 smithi103 ceph-mon[32635]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:22.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:22.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:22.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:22.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:22.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:22 smithi204 ceph-mon[32618]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:23.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:23 smithi103 ceph-mon[32635]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T20:03:23.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:23 smithi204 ceph-mon[32618]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T20:03:23.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:23 smithi094 ceph-mon[28050]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T20:03:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.065+0000 7f73b9600700 1 -- 172.21.15.94:0/3275513041 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f73bc063860 con 0x7f73a40a3cc0 2024-04-26T20:03:24.067 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 3 on host 'smithi094' 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f73a40a3cc0 msgr2=0x7f73a40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f73a40a3cc0 0x7f73a40a6180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009680 tx=0x7f73a8009260 comp rx=0 tx=0).stop 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73bc1052d0 msgr2=0x7f73bc19b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73bc1052d0 0x7f73bc19b7f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f73b000e9f0 tx=0x7f73b000ed00 comp rx=0 tx=0).stop 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 shutdown_connections 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f73a40a3cc0 0x7f73a40a6180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73bc106d90 0x7f73bc1a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73bc1052d0 0x7f73bc19b7f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 --2- 172.21.15.94:0/3275513041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc105c70 0x7f73bc19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:24.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 >> 172.21.15.94:0/3275513041 conn(0x7f73bc100c70 msgr2=0x7f73bc109b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:03:24.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.069+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 shutdown_connections 2024-04-26T20:03:24.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:03:24.070+0000 7f73c3a00700 1 -- 172.21.15.94:0/3275513041 wait complete. 2024-04-26T20:03:24.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:24 smithi204 ceph-mon[32618]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:24 smithi204 ceph-mon[32618]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T20:03:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:24 smithi204 ceph-mon[32618]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T20:03:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:24 smithi204 ceph-mon[32618]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:03:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:24 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:24 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:24 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:24 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:24.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:24 smithi094 ceph-mon[28050]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:24.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:24 smithi094 ceph-mon[28050]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T20:03:24.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:24 smithi094 ceph-mon[28050]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T20:03:24.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:24 smithi094 ceph-mon[28050]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:03:24.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:24 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:24.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:24 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:24.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:24 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:24.937 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:24 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:24.996 DEBUG:teuthology.orchestra.run.smithi094:osd.3> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T20:03:25.001 INFO:tasks.cephadm:Deploying osd.4 on smithi103 with /dev/vg_nvme/lv_3... 2024-04-26T20:03:25.001 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T20:03:25.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:24 smithi103 ceph-mon[32635]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:25.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:24 smithi103 ceph-mon[32635]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T20:03:25.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:24 smithi103 ceph-mon[32635]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T20:03:25.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:24 smithi103 ceph-mon[32635]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:03:25.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:24 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:25.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:24 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:25.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:24 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:25.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:24 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:25.246 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:03:25.683 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:25 smithi103 ceph-mon[32635]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:03:25.683 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:25 smithi103 ceph-mon[32635]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T20:03:25.683 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:25 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:25.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:25 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:25.800 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:25 smithi204 ceph-mon[32618]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:03:25.800 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:25 smithi204 ceph-mon[32618]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T20:03:25.800 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:25 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:25.800 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:25 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:25 smithi094 ceph-mon[28050]: from='osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:03:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:25 smithi094 ceph-mon[28050]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T20:03:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:25 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:25 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:26.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:26 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:03:26.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:26 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:03:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:26 smithi094 ceph-mon[28050]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:26 smithi094 ceph-mon[28050]: osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] boot 2024-04-26T20:03:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:26 smithi094 ceph-mon[28050]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T20:03:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:26.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:26 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:03:26.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:26 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:03:26.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:26 smithi103 ceph-mon[32635]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:26.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:26 smithi103 ceph-mon[32635]: osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] boot 2024-04-26T20:03:26.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:26 smithi103 ceph-mon[32635]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T20:03:26.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:26.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:26 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:03:26.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:26 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:03:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:26 smithi204 ceph-mon[32618]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:03:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:26 smithi204 ceph-mon[32618]: osd.3 [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] boot 2024-04-26T20:03:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:26 smithi204 ceph-mon[32618]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T20:03:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:03:27.089 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T20:03:27.089 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T20:03:27.089 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-26T20:03:27.089 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-26T20:03:27.089 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112691 s, 930 MB/s 2024-04-26T20:03:27.089 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-26T20:03:27.737 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:27 smithi103 ceph-mon[32635]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T20:03:27.737 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:27 smithi103 ceph-mon[32635]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T20:03:27.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_3 2024-04-26T20:03:27.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:27 smithi204 ceph-mon[32618]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T20:03:27.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:27 smithi204 ceph-mon[32618]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T20:03:27.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:27 smithi094 ceph-mon[28050]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T20:03:27.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:27 smithi094 ceph-mon[28050]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T20:03:28.042 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:03:28.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:28 smithi103 ceph-mon[32635]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:28.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:28 smithi204 ceph-mon[32618]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:28.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:28 smithi094 ceph-mon[28050]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 -- 172.21.15.103:0/2691292220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30103f10 msgr2=0x7f5c30106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 --2- 172.21.15.103:0/2691292220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30103f10 0x7f5c30106300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009990 tx=0x7f5c2c009ca0 comp rx=0 tx=0).stop 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 -- 172.21.15.103:0/2691292220 shutdown_connections 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 --2- 172.21.15.103:0/2691292220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c30109170 0x7f5c3010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 --2- 172.21.15.103:0/2691292220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30103f10 0x7f5c30106300 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 --2- 172.21.15.103:0/2691292220 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c30106840 0x7f5c30108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 -- 172.21.15.103:0/2691292220 >> 172.21.15.103:0/2691292220 conn(0x7f5c300fd940 msgr2=0x7f5c300ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 -- 172.21.15.103:0/2691292220 shutdown_connections 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 -- 172.21.15.103:0/2691292220 wait complete. 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.533+0000 7f5c37384700 1 Processor -- start 2024-04-26T20:03:29.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c37384700 1 -- start start 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c37384700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c30103f10 0x7f5c301995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c37384700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c30106840 0x7f5c30199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c37384700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30109170 0x7f5c301a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c37384700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3010ef80 con 0x7f5c30106840 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c37384700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3010ee00 con 0x7f5c30109170 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c37384700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3010f100 con 0x7f5c30103f10 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35921700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30109170 0x7f5c301a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35921700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30109170 0x7f5c301a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36802/0 (socket says 172.21.15.103:36802) 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35921700 1 -- 172.21.15.103:0/4126734571 learned_addr learned my addr 172.21.15.103:0/4126734571 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35120700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c30103f10 0x7f5c301995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35921700 1 -- 172.21.15.103:0/4126734571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c30103f10 msgr2=0x7f5c301995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35921700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c30103f10 0x7f5c301995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:29.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35921700 1 -- 172.21.15.103:0/4126734571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c30106840 msgr2=0x7f5c30199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:29.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35921700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c30106840 0x7f5c30199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:29.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35921700 1 -- 172.21.15.103:0/4126734571 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c301a1250 con 0x7f5c30109170 2024-04-26T20:03:29.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c35921700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30109170 0x7f5c301a0bb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800b9e0 tx=0x7f5c2800bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:03:29.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c267fc700 1 -- 172.21.15.103:0/4126734571 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28019070 con 0x7f5c30109170 2024-04-26T20:03:29.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.534+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c301a1540 con 0x7f5c30109170 2024-04-26T20:03:29.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.535+0000 7f5c267fc700 1 -- 172.21.15.103:0/4126734571 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c28014440 con 0x7f5c30109170 2024-04-26T20:03:29.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.535+0000 7f5c267fc700 1 -- 172.21.15.103:0/4126734571 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2800ce60 con 0x7f5c30109170 2024-04-26T20:03:29.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.535+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c301a1b00 con 0x7f5c30109170 2024-04-26T20:03:29.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.535+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c3006a110 con 0x7f5c30109170 2024-04-26T20:03:29.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.537+0000 7f5c267fc700 1 -- 172.21.15.103:0/4126734571 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5c2801c070 con 0x7f5c30109170 2024-04-26T20:03:29.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.537+0000 7f5c267fc700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5c1c0a3c40 0x7f5c1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:03:29.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.537+0000 7f5c267fc700 1 -- 172.21.15.103:0/4126734571 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f5c280cea80 con 0x7f5c30109170 2024-04-26T20:03:29.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.538+0000 7f5c35120700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5c1c0a3c40 0x7f5c1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:03:29.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.539+0000 7f5c35120700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5c1c0a3c40 0x7f5c1c0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00b2e0 tx=0x7f5c2c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:03:29.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.541+0000 7f5c267fc700 1 -- 172.21.15.103:0/4126734571 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5c2809c680 con 0x7f5c30109170 2024-04-26T20:03:29.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:29.751+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f5c30063860 con 0x7f5c1c0a3c40 2024-04-26T20:03:29.752 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:29 smithi103 ceph-mon[32635]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T20:03:29.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:29 smithi204 ceph-mon[32618]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T20:03:29.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:29 smithi094 ceph-mon[28050]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T20:03:30.718 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:30.718 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T20:03:30.718 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.718 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.718 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:03:30.718 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:30.718 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:30.718 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:03:30.719 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:30.719 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.719 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:30.719 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:30.719 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T20:03:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:03:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:03:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:31.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:31 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi094 2024-04-26T20:03:31.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:31 smithi204 ceph-mon[32618]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:03:31.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:31 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi094 to 8544M 2024-04-26T20:03:31.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:31 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi094 2024-04-26T20:03:31.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:31 smithi094 ceph-mon[28050]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:03:31.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:31 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi094 to 8544M 2024-04-26T20:03:32.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:31 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi094 2024-04-26T20:03:32.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:31 smithi103 ceph-mon[32635]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:03:32.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:31 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi094 to 8544M 2024-04-26T20:03:32.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:32 smithi103 ceph-mon[32635]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-26T20:03:32.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:32 smithi103 ceph-mon[32635]: from='client.? 172.21.15.103:0/141782504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d3851f-30c6-4ec7-a8da-0b949c46c6c8"}]: dispatch 2024-04-26T20:03:32.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:32 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d3851f-30c6-4ec7-a8da-0b949c46c6c8"}]: dispatch 2024-04-26T20:03:32.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:32 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5d3851f-30c6-4ec7-a8da-0b949c46c6c8"}]': finished 2024-04-26T20:03:32.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:32 smithi103 ceph-mon[32635]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T20:03:32.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:32 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:32.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:32 smithi204 ceph-mon[32618]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-26T20:03:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:32 smithi204 ceph-mon[32618]: from='client.? 172.21.15.103:0/141782504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d3851f-30c6-4ec7-a8da-0b949c46c6c8"}]: dispatch 2024-04-26T20:03:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:32 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d3851f-30c6-4ec7-a8da-0b949c46c6c8"}]: dispatch 2024-04-26T20:03:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:32 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5d3851f-30c6-4ec7-a8da-0b949c46c6c8"}]': finished 2024-04-26T20:03:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:32 smithi204 ceph-mon[32618]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T20:03:32.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:32.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:32 smithi094 ceph-mon[28050]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-26T20:03:32.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:32 smithi094 ceph-mon[28050]: from='client.? 172.21.15.103:0/141782504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d3851f-30c6-4ec7-a8da-0b949c46c6c8"}]: dispatch 2024-04-26T20:03:32.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:32 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d3851f-30c6-4ec7-a8da-0b949c46c6c8"}]: dispatch 2024-04-26T20:03:32.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:32 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5d3851f-30c6-4ec7-a8da-0b949c46c6c8"}]': finished 2024-04-26T20:03:32.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:32 smithi094 ceph-mon[28050]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T20:03:32.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:32 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:33.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:33 smithi204 ceph-mon[32618]: from='client.? 172.21.15.103:0/1027431577' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:03:33.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:33 smithi094 ceph-mon[28050]: from='client.? 172.21.15.103:0/1027431577' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:03:34.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:33 smithi103 ceph-mon[32635]: from='client.? 172.21.15.103:0/1027431577' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:03:34.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:34 smithi204 ceph-mon[32618]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-26T20:03:34.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:34 smithi094 ceph-mon[28050]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-26T20:03:35.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:34 smithi103 ceph-mon[32635]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-26T20:03:36.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:36 smithi103 ceph-mon[32635]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-26T20:03:36.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:36 smithi204 ceph-mon[32618]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-26T20:03:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:36 smithi094 ceph-mon[28050]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-26T20:03:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:38 smithi204 ceph-mon[32618]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:03:38.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:38 smithi094 ceph-mon[28050]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:03:39.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:38 smithi103 ceph-mon[32635]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:03:40.686 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:40 smithi103 ceph-mon[32635]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:03:40.686 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T20:03:40.686 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:40.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:40 smithi204 ceph-mon[32618]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:03:40.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T20:03:40.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:40.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:40 smithi094 ceph-mon[28050]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:03:40.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T20:03:40.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:41.900 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:41 smithi103 ceph-mon[32635]: Deploying daemon osd.4 on smithi103 2024-04-26T20:03:41.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:41 smithi094 ceph-mon[28050]: Deploying daemon osd.4 on smithi103 2024-04-26T20:03:42.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:41 smithi204 ceph-mon[32618]: Deploying daemon osd.4 on smithi103 2024-04-26T20:03:42.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:42 smithi094 ceph-mon[28050]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:43.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:42 smithi103 ceph-mon[32635]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:43.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:42 smithi204 ceph-mon[32618]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:44.713 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:44 smithi103 ceph-mon[32635]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:44.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:44 smithi094 ceph-mon[28050]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:45.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:44 smithi204 ceph-mon[32618]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:45.944 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:45.944 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:45.944 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:45.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:45.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:45.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:46.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:46.898 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:46 smithi103 ceph-mon[32635]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:46.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:46 smithi094 ceph-mon[28050]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:47.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:46 smithi204 ceph-mon[32618]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:48.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.847+0000 7f5c267fc700 1 -- 172.21.15.103:0/4126734571 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5c30063860 con 0x7f5c1c0a3c40 2024-04-26T20:03:48.848 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 4 on host 'smithi103' 2024-04-26T20:03:48.849 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:48 smithi103 ceph-mon[32635]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:48.849 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:48.849 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:48.849 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:48.849 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:48.849 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:48.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5c1c0a3c40 msgr2=0x7f5c1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5c1c0a3c40 0x7f5c1c0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00b2e0 tx=0x7f5c2c01a040 comp rx=0 tx=0).stop 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30109170 msgr2=0x7f5c301a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30109170 0x7f5c301a0bb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800b9e0 tx=0x7f5c2800bda0 comp rx=0 tx=0).stop 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 shutdown_connections 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5c1c0a3c40 0x7f5c1c0a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c30109170 0x7f5c301a0bb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c30103f10 0x7f5c301995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 --2- 172.21.15.103:0/4126734571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c30106840 0x7f5c30199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 >> 172.21.15.103:0/4126734571 conn(0x7f5c300fd940 msgr2=0x7f5c30104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 shutdown_connections 2024-04-26T20:03:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:03:48.849+0000 7f5c37384700 1 -- 172.21.15.103:0/4126734571 wait complete. 2024-04-26T20:03:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:48 smithi094 ceph-mon[28050]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:48.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:48.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:49.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:48 smithi204 ceph-mon[32618]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:49.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:49.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:49.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:49.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:49.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:49.555 DEBUG:teuthology.orchestra.run.smithi103:osd.4> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T20:03:49.557 INFO:tasks.cephadm:Deploying osd.5 on smithi204 with /dev/vg_nvme/lv_3... 2024-04-26T20:03:49.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T20:03:49.805 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:03:49.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:49.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:49.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:49.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:49 smithi204 ceph-mon[32618]: from='osd.4 [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:03:49.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:49 smithi204 ceph-mon[32618]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:03:49.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:49 smithi204 ceph-mon[32618]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:50.133 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:50.133 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:50.133 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:50.133 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:49 smithi103 ceph-mon[32635]: from='osd.4 [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:03:50.133 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:49 smithi103 ceph-mon[32635]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:03:50.133 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:49 smithi103 ceph-mon[32635]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:03:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:49 smithi094 ceph-mon[28050]: from='osd.4 [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:03:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:49 smithi094 ceph-mon[28050]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:03:50.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:49 smithi094 ceph-mon[28050]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:51.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:50 smithi204 ceph-mon[32618]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T20:03:51.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:50 smithi204 ceph-mon[32618]: from='osd.4 [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:03:51.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:50 smithi204 ceph-mon[32618]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T20:03:51.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:51.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:50 smithi204 ceph-mon[32618]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:03:51.138 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:50 smithi103 ceph-mon[32635]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T20:03:51.138 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:50 smithi103 ceph-mon[32635]: from='osd.4 [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:03:51.138 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:50 smithi103 ceph-mon[32635]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T20:03:51.138 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:51.138 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:50 smithi103 ceph-mon[32635]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:03:51.139 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:03:50 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4[43736]: 2024-04-26T20:03:50.857+0000 7f75d934a700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T20:03:51.139 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:03:50 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4[43736]: 2024-04-26T20:03:50.863+0000 7f75d513d700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:03:51.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:50 smithi094 ceph-mon[28050]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T20:03:51.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:50 smithi094 ceph-mon[28050]: from='osd.4 [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:03:51.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:50 smithi094 ceph-mon[28050]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T20:03:51.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:51.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:50 smithi094 ceph-mon[28050]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:03:51.851 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T20:03:51.851 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T20:03:51.851 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-04-26T20:03:51.851 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-04-26T20:03:51.851 INFO:teuthology.orchestra.run.smithi204.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01278 s, 820 MB/s 2024-04-26T20:03:51.851 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-04-26T20:03:52.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:51 smithi204 ceph-mon[32618]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:03:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:51 smithi204 ceph-mon[32618]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T20:03:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:51 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:51 smithi204 ceph-mon[32618]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:52.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:51 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:51 smithi094 ceph-mon[28050]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:03:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:51 smithi094 ceph-mon[28050]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T20:03:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:51 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:51 smithi094 ceph-mon[28050]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:51 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:52.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:51 smithi103 ceph-mon[32635]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:03:52.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:51 smithi103 ceph-mon[32635]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T20:03:52.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:51 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:52.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:51 smithi103 ceph-mon[32635]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T20:03:52.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:51 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:52.527 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_3 2024-04-26T20:03:52.775 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:03:53.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:03:53.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:03:53.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: osd.4 [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] boot 2024-04-26T20:03:53.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T20:03:53.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:53.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi103 2024-04-26T20:03:53.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:53.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:53.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi103 to 8544M 2024-04-26T20:03:53.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:53.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:53.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: osd.4 [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] boot 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi103 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi103 to 8544M 2024-04-26T20:03:53.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:53.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:53.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:03:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:03:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: osd.4 [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] boot 2024-04-26T20:03:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T20:03:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi103 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi103 to 8544M 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:03:53.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:03:54.050 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:53 smithi204 ceph-mon[32618]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T20:03:54.050 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:53 smithi204 ceph-mon[32618]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:03:54.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:53 smithi094 ceph-mon[28050]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T20:03:54.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:53 smithi094 ceph-mon[28050]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:03:54.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.279+0000 7f87bf4cc700 1 -- 172.21.15.204:0/1803777797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8101af0 msgr2=0x7f87b8101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.279+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/1803777797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8101af0 0x7f87b8101f10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b0a0 tx=0x7f87b400b3b0 comp rx=0 tx=0).stop 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.279+0000 7f87bf4cc700 1 -- 172.21.15.204:0/1803777797 shutdown_connections 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.279+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/1803777797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87b8103690 0x7f87b8109f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.279+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/1803777797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8101af0 0x7f87b8101f10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.279+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/1803777797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87b8102cf0 0x7f87b8103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.279+0000 7f87bf4cc700 1 -- 172.21.15.204:0/1803777797 >> 172.21.15.204:0/1803777797 conn(0x7f87b80fd0b0 msgr2=0x7f87b80ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.279+0000 7f87bf4cc700 1 -- 172.21.15.204:0/1803777797 shutdown_connections 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.279+0000 7f87bf4cc700 1 -- 172.21.15.204:0/1803777797 wait complete. 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bf4cc700 1 Processor -- start 2024-04-26T20:03:54.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bf4cc700 1 -- start start 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bf4cc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87b8101af0 0x7f87b8199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bf4cc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87b8102cf0 0x7f87b8199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bf4cc700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8103690 0x7f87b81a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bf4cc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f87b810df30 con 0x7f87b8102cf0 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bf4cc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87b810ddb0 con 0x7f87b8101af0 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bf4cc700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f87b810e0b0 con 0x7f87b8103690 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bda69700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8103690 0x7f87b81a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bda69700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8103690 0x7f87b81a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:35384/0 (socket says 172.21.15.204:35384) 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bda69700 1 -- 172.21.15.204:0/2045819557 learned_addr learned my addr 172.21.15.204:0/2045819557 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bd268700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87b8101af0 0x7f87b8199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:03:54.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bda69700 1 -- 172.21.15.204:0/2045819557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87b8101af0 msgr2=0x7f87b8199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bda69700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87b8101af0 0x7f87b8199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bda69700 1 -- 172.21.15.204:0/2045819557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87b8102cf0 msgr2=0x7f87b8199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bda69700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87b8102cf0 0x7f87b8199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bda69700 1 -- 172.21.15.204:0/2045819557 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b81a13f0 con 0x7f87b8103690 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.280+0000 7f87bd268700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87b8101af0 0x7f87b8199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.281+0000 7f87bda69700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8103690 0x7f87b81a0d50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f87b000b580 tx=0x7f87b000b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.281+0000 7f87ae7fc700 1 -- 172.21.15.204:0/2045819557 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0011830 con 0x7f87b8103690 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.281+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b81a16e0 con 0x7f87b8103690 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.281+0000 7f87ae7fc700 1 -- 172.21.15.204:0/2045819557 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b0016070 con 0x7f87b8103690 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.281+0000 7f87ae7fc700 1 -- 172.21.15.204:0/2045819557 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0010d40 con 0x7f87b8103690 2024-04-26T20:03:54.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.281+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b81a1ca0 con 0x7f87b8103690 2024-04-26T20:03:54.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:53 smithi103 ceph-mon[32635]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T20:03:54.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:53 smithi103 ceph-mon[32635]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:03:54.287 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.282+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87b806a110 con 0x7f87b8103690 2024-04-26T20:03:54.287 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.283+0000 7f87ae7fc700 1 -- 172.21.15.204:0/2045819557 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f87b00119d0 con 0x7f87b8103690 2024-04-26T20:03:54.287 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.284+0000 7f87ae7fc700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f87a40a3c40 0x7f87a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:03:54.287 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.284+0000 7f87ae7fc700 1 -- 172.21.15.204:0/2045819557 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f87b00ce4c0 con 0x7f87b8103690 2024-04-26T20:03:54.287 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.284+0000 7f87bd268700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f87a40a3c40 0x7f87a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:03:54.287 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.285+0000 7f87bd268700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f87a40a3c40 0x7f87a40a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009fd0 tx=0x7f87b4006000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:03:54.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.288+0000 7f87ae7fc700 1 -- 172.21.15.204:0/2045819557 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f87b0020060 con 0x7f87b8103690 2024-04-26T20:03:54.495 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:03:54.494+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f87b810b060 con 0x7f87a40a3c40 2024-04-26T20:03:55.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:54 smithi204 ceph-mon[32618]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:03:55.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:03:55.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:03:55.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:54 smithi094 ceph-mon[28050]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:03:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:03:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:03:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:55.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:54 smithi103 ceph-mon[32635]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:03:55.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:03:55.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:03:55.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:03:56.049 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:55 smithi204 ceph-mon[32618]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:03:56.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:55 smithi094 ceph-mon[28050]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:03:56.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:55 smithi103 ceph-mon[32635]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:03:57.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:57 smithi103 ceph-mon[32635]: from='client.? 172.21.15.204:0/859364250' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e5aa68-5a1c-449c-b613-e00d934eb78e"}]: dispatch 2024-04-26T20:03:57.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:57 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e5aa68-5a1c-449c-b613-e00d934eb78e"}]: dispatch 2024-04-26T20:03:57.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:57 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3e5aa68-5a1c-449c-b613-e00d934eb78e"}]': finished 2024-04-26T20:03:57.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:57 smithi103 ceph-mon[32635]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T20:03:57.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:03:57.556 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:57 smithi204 ceph-mon[32618]: from='client.? 172.21.15.204:0/859364250' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e5aa68-5a1c-449c-b613-e00d934eb78e"}]: dispatch 2024-04-26T20:03:57.556 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:57 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e5aa68-5a1c-449c-b613-e00d934eb78e"}]: dispatch 2024-04-26T20:03:57.556 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:57 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3e5aa68-5a1c-449c-b613-e00d934eb78e"}]': finished 2024-04-26T20:03:57.556 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:57 smithi204 ceph-mon[32618]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T20:03:57.557 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:03:57.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:57 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/859364250' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e5aa68-5a1c-449c-b613-e00d934eb78e"}]: dispatch 2024-04-26T20:03:57.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:57 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e5aa68-5a1c-449c-b613-e00d934eb78e"}]: dispatch 2024-04-26T20:03:57.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:57 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3e5aa68-5a1c-449c-b613-e00d934eb78e"}]': finished 2024-04-26T20:03:57.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:57 smithi094 ceph-mon[28050]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T20:03:57.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:03:58.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:58 smithi103 ceph-mon[32635]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:03:58.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:03:58 smithi103 ceph-mon[32635]: from='client.? 172.21.15.204:0/79073111' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:03:58.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:58 smithi204 ceph-mon[32618]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:03:58.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:03:58 smithi204 ceph-mon[32618]: from='client.? 172.21.15.204:0/79073111' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:03:58.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:58 smithi094 ceph-mon[28050]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:03:58.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:03:58 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/79073111' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:04:00.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:00 smithi103 ceph-mon[32635]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:00.840 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:00 smithi204 ceph-mon[32618]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:00.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:00 smithi094 ceph-mon[28050]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:02.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:02 smithi204 ceph-mon[32618]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:02.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:02 smithi094 ceph-mon[28050]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:03.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:02 smithi103 ceph-mon[32635]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:04.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:04 smithi204 ceph-mon[32618]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:04.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T20:04:04.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:04.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:04 smithi094 ceph-mon[28050]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:04.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T20:04:04.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:05.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:04 smithi103 ceph-mon[32635]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:05.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T20:04:05.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:05.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:05 smithi204 ceph-mon[32618]: Deploying daemon osd.5 on smithi204 2024-04-26T20:04:05.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:05 smithi094 ceph-mon[28050]: Deploying daemon osd.5 on smithi204 2024-04-26T20:04:06.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:05 smithi103 ceph-mon[32635]: Deploying daemon osd.5 on smithi204 2024-04-26T20:04:06.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:06 smithi204 ceph-mon[32618]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:06.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:06 smithi094 ceph-mon[28050]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:07.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:06 smithi103 ceph-mon[32635]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:08.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:08 smithi204 ceph-mon[32618]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:08.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:08 smithi094 ceph-mon[28050]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:09.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:08 smithi103 ceph-mon[32635]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:10.594 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:10 smithi204 ceph-mon[32618]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:10.594 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:10.595 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:10.595 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:10 smithi094 ceph-mon[28050]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:10 smithi103 ceph-mon[32635]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:12.738 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:12 smithi204 ceph-mon[32618]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:12.836 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:12 smithi103 ceph-mon[32635]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:12 smithi094 ceph-mon[28050]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:13.728 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.727+0000 7f87ae7fc700 1 -- 172.21.15.204:0/2045819557 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f87b810b060 con 0x7f87a40a3c40 2024-04-26T20:04:13.728 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 5 on host 'smithi204' 2024-04-26T20:04:13.730 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f87a40a3c40 msgr2=0x7f87a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:13.730 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f87a40a3c40 0x7f87a40a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009fd0 tx=0x7f87b4006000 comp rx=0 tx=0).stop 2024-04-26T20:04:13.730 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8103690 msgr2=0x7f87b81a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:13.730 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8103690 0x7f87b81a0d50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f87b000b580 tx=0x7f87b000b940 comp rx=0 tx=0).stop 2024-04-26T20:04:13.730 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 shutdown_connections 2024-04-26T20:04:13.730 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87b8103690 0x7f87b81a0d50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:13.730 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f87a40a3c40 0x7f87a40a6100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:13.731 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87b8101af0 0x7f87b8199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:13.731 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 --2- 172.21.15.204:0/2045819557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87b8102cf0 0x7f87b8199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:13.731 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 >> 172.21.15.204:0/2045819557 conn(0x7f87b80fd0b0 msgr2=0x7f87b810a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:04:13.731 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 shutdown_connections 2024-04-26T20:04:13.731 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:04:13.729+0000 7f87bf4cc700 1 -- 172.21.15.204:0/2045819557 wait complete. 2024-04-26T20:04:14.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='osd.5 [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:04:14.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:13 smithi204 ceph-mon[32618]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:14.223 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='osd.5 [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:04:14.224 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:13 smithi103 ceph-mon[32635]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:04:14.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:14.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:14.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:14.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:14.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:14.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='osd.5 [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:04:14.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:13 smithi094 ceph-mon[28050]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:04:14.356 DEBUG:teuthology.orchestra.run.smithi204:osd.5> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T20:04:14.358 INFO:tasks.cephadm:Deploying osd.6 on smithi094 with /dev/vg_nvme/lv_2... 2024-04-26T20:04:14.358 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T20:04:14.797 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:04:15.784 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:15 smithi204 ceph-mon[32618]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T20:04:15.784 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:15 smithi204 ceph-mon[32618]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T20:04:15.784 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:15.785 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:15 smithi204 ceph-mon[32618]: from='osd.5 [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:04:15.785 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:15 smithi204 ceph-mon[32618]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:04:15.785 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:15 smithi204 ceph-mon[32618]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:15.785 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:04:15 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:04:15.726+0000 7f85749bf700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T20:04:15.785 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:04:15 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:04:15.732+0000 7f85707b2700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:04:15.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:15 smithi094 ceph-mon[28050]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T20:04:15.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:15 smithi094 ceph-mon[28050]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T20:04:15.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:15.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:15 smithi094 ceph-mon[28050]: from='osd.5 [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:04:15.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:15 smithi094 ceph-mon[28050]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:04:15.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:15 smithi094 ceph-mon[28050]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:16.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:15 smithi103 ceph-mon[32635]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T20:04:16.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:15 smithi103 ceph-mon[32635]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T20:04:16.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:16.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:15 smithi103 ceph-mon[32635]: from='osd.5 [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:04:16.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:15 smithi103 ceph-mon[32635]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:04:16.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:15 smithi103 ceph-mon[32635]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:16.826 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:16 smithi204 ceph-mon[32618]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:04:16.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:16 smithi204 ceph-mon[32618]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T20:04:16.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:16 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:16.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:16 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:16.866 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:16 smithi094 ceph-mon[28050]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:04:16.866 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:16 smithi094 ceph-mon[28050]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T20:04:16.866 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:16 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:16.866 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:16 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:17.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:16 smithi103 ceph-mon[32635]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:04:17.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:16 smithi103 ceph-mon[32635]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T20:04:17.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:16 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:17.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:16 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:17.914 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T20:04:17.914 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T20:04:17.914 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-04-26T20:04:17.914 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-04-26T20:04:17.914 INFO:teuthology.orchestra.run.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0357041 s, 294 MB/s 2024-04-26T20:04:17.914 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-04-26T20:04:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:04:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:04:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: osd.5 [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] boot 2024-04-26T20:04:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T20:04:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi204 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi204 to 8544M 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:17 smithi094 ceph-mon[28050]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:18.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:04:18.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:04:18.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: osd.5 [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] boot 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi204 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi204 to 8544M 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:18.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:17 smithi103 ceph-mon[32635]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: osd.5 [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] boot 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi204 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:18.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:18.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:18.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi204 to 8544M 2024-04-26T20:04:18.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:18.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:18.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:18.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:18.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:17 smithi204 ceph-mon[32618]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:04:18.994 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:18 smithi094 ceph-mon[28050]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T20:04:18.999 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_2 2024-04-26T20:04:19.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:18 smithi103 ceph-mon[32635]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T20:04:19.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:18 smithi204 ceph-mon[32618]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T20:04:19.430 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:04:19.912 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:19 smithi094 ceph-mon[28050]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T20:04:19.913 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:19 smithi094 ceph-mon[28050]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T20:04:19.913 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:19 smithi094 ceph-mon[28050]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:20.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:19 smithi103 ceph-mon[32635]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T20:04:20.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:19 smithi103 ceph-mon[32635]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T20:04:20.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:19 smithi103 ceph-mon[32635]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:20.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:19 smithi204 ceph-mon[32618]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T20:04:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:19 smithi204 ceph-mon[32618]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T20:04:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:19 smithi204 ceph-mon[32618]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:21.653 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:21 smithi094 ceph-mon[28050]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T20:04:21.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:21 smithi103 ceph-mon[32635]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T20:04:21.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.822+0000 7f12aa000700 1 -- 172.21.15.94:0/2962072267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 msgr2=0x7f12a4076300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:21.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.822+0000 7f12aa000700 1 --2- 172.21.15.94:0/2962072267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 0x7f12a4076300 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1294009990 tx=0x7f1294009ca0 comp rx=0 tx=0).stop 2024-04-26T20:04:21.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.823+0000 7f12aa000700 1 -- 172.21.15.94:0/2962072267 shutdown_connections 2024-04-26T20:04:21.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.823+0000 7f12aa000700 1 --2- 172.21.15.94:0/2962072267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12a410b270 0x7f12a410d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:21.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.823+0000 7f12aa000700 1 --2- 172.21.15.94:0/2962072267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12a4077450 0x7f12a40758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:21.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.823+0000 7f12aa000700 1 --2- 172.21.15.94:0/2962072267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 0x7f12a4076300 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:21.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.823+0000 7f12aa000700 1 -- 172.21.15.94:0/2962072267 >> 172.21.15.94:0/2962072267 conn(0x7f12a40fe960 msgr2=0x7f12a4100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:04:21.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.823+0000 7f12aa000700 1 -- 172.21.15.94:0/2962072267 shutdown_connections 2024-04-26T20:04:21.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.823+0000 7f12aa000700 1 -- 172.21.15.94:0/2962072267 wait complete. 2024-04-26T20:04:21.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.824+0000 7f12aa000700 1 Processor -- start 2024-04-26T20:04:21.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.824+0000 7f12aa000700 1 -- start start 2024-04-26T20:04:21.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.824+0000 7f12aa000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 0x7f12a419b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:04:21.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.824+0000 7f12aa000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12a4077450 0x7f12a419bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:04:21.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.824+0000 7f12aa000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12a410b270 0x7f12a41a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.824+0000 7f12aa000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4111080 con 0x7f12a4075e80 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.824+0000 7f12aa000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4110f00 con 0x7f12a410b270 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.824+0000 7f12aa000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4111200 con 0x7f12a4077450 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a8c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 0x7f12a419b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a8c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 0x7f12a419b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59714/0 (socket says 172.21.15.94:59714) 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a8c00700 1 -- 172.21.15.94:0/247936153 learned_addr learned my addr 172.21.15.94:0/247936153 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a3e00700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12a4077450 0x7f12a419bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a9600700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12a410b270 0x7f12a41a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a8c00700 1 -- 172.21.15.94:0/247936153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12a4077450 msgr2=0x7f12a419bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a8c00700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12a4077450 0x7f12a419bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a8c00700 1 -- 172.21.15.94:0/247936153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12a410b270 msgr2=0x7f12a41a2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a8c00700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12a410b270 0x7f12a41a2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:21.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a8c00700 1 -- 172.21.15.94:0/247936153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a41a3390 con 0x7f12a4075e80 2024-04-26T20:04:21.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.825+0000 7f12a8c00700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 0x7f12a419b780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f129800e9a0 tx=0x7f129800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:04:21.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.826+0000 7f12a1600700 1 -- 172.21.15.94:0/247936153 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298004a30 con 0x7f12a4075e80 2024-04-26T20:04:21.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.826+0000 7f12a1600700 1 -- 172.21.15.94:0/247936153 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f129800ccc0 con 0x7f12a4075e80 2024-04-26T20:04:21.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.826+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a41a3680 con 0x7f12a4075e80 2024-04-26T20:04:21.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.826+0000 7f12a1600700 1 -- 172.21.15.94:0/247936153 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298005610 con 0x7f12a4075e80 2024-04-26T20:04:21.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.826+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a41a3c10 con 0x7f12a4075e80 2024-04-26T20:04:21.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.827+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12a406a110 con 0x7f12a4075e80 2024-04-26T20:04:21.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.829+0000 7f12a1600700 1 -- 172.21.15.94:0/247936153 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f129800c810 con 0x7f12a4075e80 2024-04-26T20:04:21.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.831+0000 7f12a1600700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f128c0a3cc0 0x7f128c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:04:21.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.831+0000 7f12a1600700 1 -- 172.21.15.94:0/247936153 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f12980cf840 con 0x7f12a4075e80 2024-04-26T20:04:21.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.832+0000 7f12a3e00700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f128c0a3cc0 0x7f128c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:04:21.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.832+0000 7f12a3e00700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f128c0a3cc0 0x7f128c0a6180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f129400b2e0 tx=0x7f1294005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:04:21.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:21.836+0000 7f12a1600700 1 -- 172.21.15.94:0/247936153 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f129809d040 con 0x7f12a4075e80 2024-04-26T20:04:21.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:21 smithi204 ceph-mon[32618]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T20:04:22.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:22.072+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f12a4063860 con 0x7f128c0a3cc0 2024-04-26T20:04:22.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:22 smithi094 ceph-mon[28050]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T20:04:22.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:04:22.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:04:22.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:22.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:22 smithi103 ceph-mon[32635]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T20:04:22.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:04:22.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:04:22.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:22.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:22 smithi204 ceph-mon[32618]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T20:04:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:04:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:04:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:23.722 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:23 smithi103 ceph-mon[32635]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:04:23.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:23 smithi094 ceph-mon[28050]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:04:23.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:23 smithi204 ceph-mon[32618]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:04:24.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:24 smithi094 ceph-mon[28050]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-26T20:04:24.783 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:24 smithi204 ceph-mon[32618]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-26T20:04:24.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:24 smithi103 ceph-mon[32635]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-26T20:04:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:26 smithi094 ceph-mon[28050]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-26T20:04:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:26 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4191219123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438ff921-8607-404a-bffd-7234940076bd"}]: dispatch 2024-04-26T20:04:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:26 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438ff921-8607-404a-bffd-7234940076bd"}]: dispatch 2024-04-26T20:04:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:26 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "438ff921-8607-404a-bffd-7234940076bd"}]': finished 2024-04-26T20:04:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:26 smithi094 ceph-mon[28050]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T20:04:26.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:26 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:26 smithi204 ceph-mon[32618]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-26T20:04:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:26 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4191219123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438ff921-8607-404a-bffd-7234940076bd"}]: dispatch 2024-04-26T20:04:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:26 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438ff921-8607-404a-bffd-7234940076bd"}]: dispatch 2024-04-26T20:04:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:26 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "438ff921-8607-404a-bffd-7234940076bd"}]': finished 2024-04-26T20:04:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:26 smithi204 ceph-mon[32618]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T20:04:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:26 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:26 smithi103 ceph-mon[32635]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-26T20:04:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:26 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4191219123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438ff921-8607-404a-bffd-7234940076bd"}]: dispatch 2024-04-26T20:04:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:26 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "438ff921-8607-404a-bffd-7234940076bd"}]: dispatch 2024-04-26T20:04:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:26 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "438ff921-8607-404a-bffd-7234940076bd"}]': finished 2024-04-26T20:04:27.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:26 smithi103 ceph-mon[32635]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T20:04:27.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:26 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:27.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:27 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2886387698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:04:27.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:27 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2886387698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:04:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:27 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2886387698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:04:28.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:28 smithi204 ceph-mon[32618]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:04:28.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:28 smithi094 ceph-mon[28050]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:04:29.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:28 smithi103 ceph-mon[32635]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:04:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:30 smithi204 ceph-mon[32618]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T20:04:30.904 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:30 smithi094 ceph-mon[28050]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T20:04:31.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:30 smithi103 ceph-mon[32635]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T20:04:32.596 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:32 smithi094 ceph-mon[28050]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:32.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:32 smithi204 ceph-mon[32618]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:33.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:32 smithi103 ceph-mon[32635]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:34.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:34 smithi204 ceph-mon[32618]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:34.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:34 smithi094 ceph-mon[28050]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:35.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:34 smithi103 ceph-mon[32635]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:35.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:35 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T20:04:35.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:35 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:35.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:35 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T20:04:35.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:35 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:36.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:35 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T20:04:36.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:35 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:36.836 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:36 smithi094 ceph-mon[28050]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:36.836 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:36 smithi094 ceph-mon[28050]: Deploying daemon osd.6 on smithi094 2024-04-26T20:04:36.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:36 smithi204 ceph-mon[32618]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:36.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:36 smithi204 ceph-mon[32618]: Deploying daemon osd.6 on smithi094 2024-04-26T20:04:37.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:36 smithi103 ceph-mon[32635]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:37.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:36 smithi103 ceph-mon[32635]: Deploying daemon osd.6 on smithi094 2024-04-26T20:04:38.862 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:38 smithi094 ceph-mon[28050]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:38.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:38 smithi204 ceph-mon[32618]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:39.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:38 smithi103 ceph-mon[32635]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:40.778 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:40 smithi094 ceph-mon[28050]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:40.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:40 smithi204 ceph-mon[32618]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:41.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:40 smithi103 ceph-mon[32635]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:42.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:42 smithi094 ceph-mon[28050]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:42.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:42 smithi204 ceph-mon[32618]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:43.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:42 smithi103 ceph-mon[32635]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:44 smithi204 ceph-mon[32618]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:44 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:44.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:44 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:44.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:44 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:44.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:44 smithi094 ceph-mon[28050]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:44.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:44 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:44.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:44 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:44.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:44 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:45.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:44 smithi103 ceph-mon[32635]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:45.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:44 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:45.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:44 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:45.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:44 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:46.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:46 smithi204 ceph-mon[32618]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:46.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:46 smithi094 ceph-mon[28050]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:47.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:46 smithi103 ceph-mon[32635]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:48.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:48 smithi204 ceph-mon[32618]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:48.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:48.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:48.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:48 smithi204 ceph-mon[32618]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T20:04:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:48 smithi094 ceph-mon[28050]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:48.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:48.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:48.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:48 smithi094 ceph-mon[28050]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T20:04:49.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:48 smithi103 ceph-mon[32635]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:49.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:49.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:49.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:48 smithi103 ceph-mon[32635]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T20:04:49.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.415+0000 7f12a1600700 1 -- 172.21.15.94:0/247936153 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f12a4063860 con 0x7f128c0a3cc0 2024-04-26T20:04:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 6 on host 'smithi094' 2024-04-26T20:04:49.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f128c0a3cc0 msgr2=0x7f128c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:49.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f128c0a3cc0 0x7f128c0a6180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f129400b2e0 tx=0x7f1294005fb0 comp rx=0 tx=0).stop 2024-04-26T20:04:49.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 msgr2=0x7f12a419b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:49.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 0x7f12a419b780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f129800e9a0 tx=0x7f129800ecb0 comp rx=0 tx=0).stop 2024-04-26T20:04:49.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 shutdown_connections 2024-04-26T20:04:49.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f128c0a3cc0 0x7f128c0a6180 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:49.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12a410b270 0x7f12a41a2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:49.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12a4075e80 0x7f12a419b780 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:49.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 --2- 172.21.15.94:0/247936153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12a4077450 0x7f12a419bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:49.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.419+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 >> 172.21.15.94:0/247936153 conn(0x7f12a40fe960 msgr2=0x7f12a4107170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:04:49.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.420+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 shutdown_connections 2024-04-26T20:04:49.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:04:49.420+0000 7f12aa000700 1 -- 172.21.15.94:0/247936153 wait complete. 2024-04-26T20:04:50.236 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:50 smithi094 ceph-mon[28050]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T20:04:50.236 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:50 smithi094 ceph-mon[28050]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T20:04:50.236 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:50 smithi094 ceph-mon[28050]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:04:50.236 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:50.236 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:50.236 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:50.236 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:50 smithi094 ceph-mon[28050]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:50.302 DEBUG:teuthology.orchestra.run.smithi094:osd.6> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T20:04:50.307 INFO:tasks.cephadm:Deploying osd.7 on smithi103 with /dev/vg_nvme/lv_2... 2024-04-26T20:04:50.307 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T20:04:50.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:50 smithi204 ceph-mon[32618]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T20:04:50.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:50 smithi204 ceph-mon[32618]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T20:04:50.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:50 smithi204 ceph-mon[32618]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:04:50.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:50.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:50.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:50.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:50 smithi204 ceph-mon[32618]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:50.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:50 smithi103 ceph-mon[32635]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T20:04:50.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:50 smithi103 ceph-mon[32635]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T20:04:50.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:50 smithi103 ceph-mon[32635]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:04:50.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:50.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:50.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:50.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:50 smithi103 ceph-mon[32635]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:50.558 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:04:50.822 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:04:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:04:50.791+0000 7f6332600700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T20:04:50.822 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:04:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:04:50.820+0000 7f6328400700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:04:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:51 smithi103 ceph-mon[32635]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:04:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:51 smithi103 ceph-mon[32635]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T20:04:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:51 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:51 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:51 smithi103 ceph-mon[32635]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' 2024-04-26T20:04:51.349 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:51 smithi094 ceph-mon[28050]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:04:51.349 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:51 smithi094 ceph-mon[28050]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T20:04:51.349 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:51 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:51.349 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:51 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:51.349 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:51 smithi094 ceph-mon[28050]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' 2024-04-26T20:04:51.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:51 smithi204 ceph-mon[32618]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:04:51.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:51 smithi204 ceph-mon[32618]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T20:04:51.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:51 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:51.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:51 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:51.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:51 smithi204 ceph-mon[32618]: from='osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603]' entity='osd.6' 2024-04-26T20:04:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:52 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:04:52.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:52 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:04:52.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:52.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:52 smithi103 ceph-mon[32635]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:52.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:52 smithi103 ceph-mon[32635]: osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] boot 2024-04-26T20:04:52.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:52 smithi103 ceph-mon[32635]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T20:04:52.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:52.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:52 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:04:52.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:52 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:04:52.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:52.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:52 smithi204 ceph-mon[32618]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:52.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:52 smithi204 ceph-mon[32618]: osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] boot 2024-04-26T20:04:52.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:52 smithi204 ceph-mon[32618]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T20:04:52.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:52.433 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:52 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:04:52.433 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:52 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:04:52.434 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:52.434 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:52 smithi094 ceph-mon[28050]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:04:52.434 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:52 smithi094 ceph-mon[28050]: osd.6 [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] boot 2024-04-26T20:04:52.434 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:52 smithi094 ceph-mon[28050]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T20:04:52.434 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:04:52.543 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T20:04:52.543 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T20:04:52.543 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-26T20:04:52.543 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-26T20:04:52.543 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114254 s, 918 MB/s 2024-04-26T20:04:52.544 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-26T20:04:53.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_2 2024-04-26T20:04:53.498 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:04:54.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:53 smithi204 ceph-mon[32618]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T20:04:54.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:53 smithi204 ceph-mon[32618]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:04:54.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:53 smithi094 ceph-mon[28050]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T20:04:54.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:53 smithi094 ceph-mon[28050]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:04:54.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:53 smithi103 ceph-mon[32635]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T20:04:54.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:53 smithi103 ceph-mon[32635]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:04:55.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.069+0000 7f2c498e6700 1 -- 172.21.15.103:0/572004800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c44103f40 msgr2=0x7f2c4410a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:55.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.069+0000 7f2c498e6700 1 --2- 172.21.15.103:0/572004800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c44103f40 0x7f2c4410a7f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00b0a0 tx=0x7f2c3c00b3b0 comp rx=0 tx=0).stop 2024-04-26T20:04:55.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.069+0000 7f2c498e6700 1 -- 172.21.15.103:0/572004800 shutdown_connections 2024-04-26T20:04:55.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.069+0000 7f2c498e6700 1 --2- 172.21.15.103:0/572004800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c44103f40 0x7f2c4410a7f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:55.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.069+0000 7f2c498e6700 1 --2- 172.21.15.103:0/572004800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c441023a0 0x7f2c441027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:55.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.069+0000 7f2c498e6700 1 --2- 172.21.15.103:0/572004800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c441035a0 0x7f2c44103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:55.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.069+0000 7f2c498e6700 1 -- 172.21.15.103:0/572004800 >> 172.21.15.103:0/572004800 conn(0x7f2c440fd940 msgr2=0x7f2c440ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:04:55.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.069+0000 7f2c498e6700 1 -- 172.21.15.103:0/572004800 shutdown_connections 2024-04-26T20:04:55.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.069+0000 7f2c498e6700 1 -- 172.21.15.103:0/572004800 wait complete. 2024-04-26T20:04:55.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c498e6700 1 Processor -- start 2024-04-26T20:04:55.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c498e6700 1 -- start start 2024-04-26T20:04:55.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c498e6700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c441023a0 0x7f2c44198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:04:55.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c498e6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c441035a0 0x7f2c44198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:04:55.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c498e6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c44103f40 0x7f2c4419fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:04:55.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c498e6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4410deb0 con 0x7f2c44103f40 2024-04-26T20:04:55.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c498e6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4410dd30 con 0x7f2c441035a0 2024-04-26T20:04:55.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c498e6700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4410e030 con 0x7f2c441023a0 2024-04-26T20:04:55.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c3a7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c441035a0 0x7f2c44198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c3a7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c441035a0 0x7f2c44198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39772/0 (socket says 172.21.15.103:39772) 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c3a7fc700 1 -- 172.21.15.103:0/1433388670 learned_addr learned my addr 172.21.15.103:0/1433388670 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.070+0000 7f2c42ffd700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c441023a0 0x7f2c44198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c3a7fc700 1 -- 172.21.15.103:0/1433388670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c441023a0 msgr2=0x7f2c44198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c3a7fc700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c441023a0 0x7f2c44198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c3a7fc700 1 -- 172.21.15.103:0/1433388670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c44103f40 msgr2=0x7f2c4419fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c3a7fc700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c44103f40 0x7f2c4419fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c3a7fc700 1 -- 172.21.15.103:0/1433388670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c441a0160 con 0x7f2c441035a0 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c42ffd700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c441023a0 0x7f2c44198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c3a7fc700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c441035a0 0x7f2c44198a40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00e9a0 tx=0x7f2c2c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c488e4700 1 -- 172.21.15.103:0/1433388670 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c00cbd0 con 0x7f2c441035a0 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c441a0450 con 0x7f2c441035a0 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c488e4700 1 -- 172.21.15.103:0/1433388670 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c2c00cd70 con 0x7f2c441035a0 2024-04-26T20:04:55.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c488e4700 1 -- 172.21.15.103:0/1433388670 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c018660 con 0x7f2c441035a0 2024-04-26T20:04:55.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.071+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c441a0a60 con 0x7f2c441035a0 2024-04-26T20:04:55.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.072+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c4406a110 con 0x7f2c441035a0 2024-04-26T20:04:55.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.073+0000 7f2c488e4700 1 -- 172.21.15.103:0/1433388670 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2c2c010430 con 0x7f2c441035a0 2024-04-26T20:04:55.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.073+0000 7f2c488e4700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c300a3c40 0x7f2c300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:04:55.077 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.074+0000 7f2c488e4700 1 -- 172.21.15.103:0/1433388670 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f2c2c0cf8e0 con 0x7f2c441035a0 2024-04-26T20:04:55.077 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.074+0000 7f2c42ffd700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c300a3c40 0x7f2c300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:04:55.077 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.075+0000 7f2c42ffd700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c300a3c40 0x7f2c300a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2c44103400 tx=0x7f2c34009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:04:55.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.078+0000 7f2c488e4700 1 -- 172.21.15.103:0/1433388670 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2c2c09cfd0 con 0x7f2c441035a0 2024-04-26T20:04:55.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:04:55.285+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2c44063860 con 0x7f2c300a3c40 2024-04-26T20:04:55.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:04:55.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:04:55.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:55.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:04:55.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:04:55.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:04:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:04:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi094 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi094 to 5696M 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:56.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:56.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:56.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:56.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:56.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:56 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.908 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:04:56.908 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:04:56.908 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi094 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi094 to 5696M 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:56.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:56 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi094 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi094 to 5696M 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:56.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:56.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:04:56.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:04:56.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:04:56.989 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:56 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:04:58.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:58 smithi103 ceph-mon[32635]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:04:58.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:58 smithi103 ceph-mon[32635]: from='client.? 172.21.15.103:0/3952895961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1df6748-fae6-4f37-806d-a6093ec9ca60"}]: dispatch 2024-04-26T20:04:58.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:58 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1df6748-fae6-4f37-806d-a6093ec9ca60"}]: dispatch 2024-04-26T20:04:58.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:58 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1df6748-fae6-4f37-806d-a6093ec9ca60"}]': finished 2024-04-26T20:04:58.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:58 smithi103 ceph-mon[32635]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T20:04:58.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:58 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:04:58.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:58 smithi204 ceph-mon[32618]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:04:58.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:58 smithi204 ceph-mon[32618]: from='client.? 172.21.15.103:0/3952895961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1df6748-fae6-4f37-806d-a6093ec9ca60"}]: dispatch 2024-04-26T20:04:58.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:58 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1df6748-fae6-4f37-806d-a6093ec9ca60"}]: dispatch 2024-04-26T20:04:58.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:58 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1df6748-fae6-4f37-806d-a6093ec9ca60"}]': finished 2024-04-26T20:04:58.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:58 smithi204 ceph-mon[32618]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T20:04:58.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:58 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:04:58.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:58 smithi094 ceph-mon[28050]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:04:58.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:58 smithi094 ceph-mon[28050]: from='client.? 172.21.15.103:0/3952895961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1df6748-fae6-4f37-806d-a6093ec9ca60"}]: dispatch 2024-04-26T20:04:58.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:58 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1df6748-fae6-4f37-806d-a6093ec9ca60"}]: dispatch 2024-04-26T20:04:58.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:58 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1df6748-fae6-4f37-806d-a6093ec9ca60"}]': finished 2024-04-26T20:04:58.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:58 smithi094 ceph-mon[28050]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T20:04:58.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:04:59.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:04:59 smithi204 ceph-mon[32618]: from='client.? 172.21.15.103:0/3625881850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:04:59.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:04:59 smithi094 ceph-mon[28050]: from='client.? 172.21.15.103:0/3625881850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:05:00.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:04:59 smithi103 ceph-mon[32635]: from='client.? 172.21.15.103:0/3625881850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:05:00.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:00 smithi204 ceph-mon[32618]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:00.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:00 smithi094 ceph-mon[28050]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:01.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:00 smithi103 ceph-mon[32635]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:02.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:02 smithi103 ceph-mon[32635]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:02.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:02 smithi204 ceph-mon[32618]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:02.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:02 smithi094 ceph-mon[28050]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:04.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:04 smithi103 ceph-mon[32635]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:04.830 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:04 smithi204 ceph-mon[32618]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:04.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:04 smithi094 ceph-mon[28050]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:05.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T20:05:05.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:05.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T20:05:05.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:05.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T20:05:05.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:06.739 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:06 smithi103 ceph-mon[32635]: Deploying daemon osd.7 on smithi103 2024-04-26T20:05:06.739 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:06 smithi103 ceph-mon[32635]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:06.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:06 smithi204 ceph-mon[32618]: Deploying daemon osd.7 on smithi103 2024-04-26T20:05:06.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:06 smithi204 ceph-mon[32618]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:06.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:06 smithi094 ceph-mon[28050]: Deploying daemon osd.7 on smithi103 2024-04-26T20:05:06.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:06 smithi094 ceph-mon[28050]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:08.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:08 smithi204 ceph-mon[32618]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:08.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:08 smithi094 ceph-mon[28050]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:09.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:08 smithi103 ceph-mon[32635]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:10 smithi204 ceph-mon[32618]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:10 smithi094 ceph-mon[28050]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:10 smithi103 ceph-mon[32635]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:12.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:12.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:12.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:12.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:11 smithi204 ceph-mon[32618]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:11 smithi094 ceph-mon[28050]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:12.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:12.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:12.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:12.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:11 smithi103 ceph-mon[32635]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:14.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.173+0000 7f2c488e4700 1 -- 172.21.15.103:0/1433388670 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2c44063860 con 0x7f2c300a3c40 2024-04-26T20:05:14.178 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 7 on host 'smithi103' 2024-04-26T20:05:14.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c300a3c40 msgr2=0x7f2c300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:14.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c300a3c40 0x7f2c300a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2c44103400 tx=0x7f2c34009190 comp rx=0 tx=0).stop 2024-04-26T20:05:14.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c441035a0 msgr2=0x7f2c44198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:14.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c441035a0 0x7f2c44198a40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00e9a0 tx=0x7f2c2c00ecb0 comp rx=0 tx=0).stop 2024-04-26T20:05:14.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 shutdown_connections 2024-04-26T20:05:14.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c44103f40 0x7f2c4419fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:14.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c441023a0 0x7f2c44198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:14.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c300a3c40 0x7f2c300a6100 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:14.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 --2- 172.21.15.103:0/1433388670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c441035a0 0x7f2c44198a40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:14.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 >> 172.21.15.103:0/1433388670 conn(0x7f2c440fd940 msgr2=0x7f2c44106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:05:14.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 shutdown_connections 2024-04-26T20:05:14.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:05:14.176+0000 7f2c498e6700 1 -- 172.21.15.103:0/1433388670 wait complete. 2024-04-26T20:05:14.674 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:14 smithi103 ceph-mon[32635]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:14.674 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:14.674 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:14.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:14 smithi204 ceph-mon[32618]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:14.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:14.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:14.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:14 smithi094 ceph-mon[28050]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:14.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:14.941 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:15.094 DEBUG:teuthology.orchestra.run.smithi103:osd.7> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T20:05:15.096 INFO:tasks.cephadm:Deploying osd.8 on smithi204 with /dev/vg_nvme/lv_2... 2024-04-26T20:05:15.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T20:05:15.342 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:05:15.715 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:15 smithi204 ceph-mon[32618]: from='osd.7 [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:05:15.715 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:15 smithi204 ceph-mon[32618]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:05:15.715 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:15.716 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:15.887 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:15 smithi103 ceph-mon[32635]: from='osd.7 [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:05:15.887 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:15 smithi103 ceph-mon[32635]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:05:15.887 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:15.887 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:15.976 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:15 smithi094 ceph-mon[28050]: from='osd.7 [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:05:15.977 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:15 smithi094 ceph-mon[28050]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:05:15.977 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:15.977 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:16.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:16 smithi204 ceph-mon[32618]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:16.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:16 smithi204 ceph-mon[32618]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T20:05:16.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:16 smithi204 ceph-mon[32618]: from='osd.7 [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:05:16.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:16 smithi204 ceph-mon[32618]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T20:05:16.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:16 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:16.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:16 smithi204 ceph-mon[32618]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:05:16.932 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:16 smithi103 ceph-mon[32635]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:16.932 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:16 smithi103 ceph-mon[32635]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T20:05:16.932 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:16 smithi103 ceph-mon[32635]: from='osd.7 [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:05:16.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:16 smithi103 ceph-mon[32635]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T20:05:16.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:16 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:16.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:16 smithi103 ceph-mon[32635]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:05:16.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:16 smithi094 ceph-mon[28050]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:16.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:16 smithi094 ceph-mon[28050]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T20:05:16.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:16 smithi094 ceph-mon[28050]: from='osd.7 [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:05:16.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:16 smithi094 ceph-mon[28050]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T20:05:16.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:16 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:16.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:16 smithi094 ceph-mon[28050]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:05:17.403 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T20:05:17.403 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T20:05:17.403 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-04-26T20:05:17.404 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-04-26T20:05:17.404 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123668 s, 848 MB/s 2024-04-26T20:05:17.404 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-04-26T20:05:17.585 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:05:17 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:05:17.326+0000 7f1b2f177700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T20:05:17.585 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:05:17 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:05:17.336+0000 7f1b26f62700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:05:17.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:17 smithi204 ceph-mon[32618]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:05:17.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:17 smithi204 ceph-mon[32618]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T20:05:17.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:17.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:17.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:17 smithi204 ceph-mon[32618]: from='osd.7 ' entity='osd.7' 2024-04-26T20:05:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:17 smithi094 ceph-mon[28050]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:05:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:17 smithi094 ceph-mon[28050]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T20:05:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:17.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:17 smithi094 ceph-mon[28050]: from='osd.7 ' entity='osd.7' 2024-04-26T20:05:18.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:17 smithi103 ceph-mon[32635]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:05:18.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:17 smithi103 ceph-mon[32635]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T20:05:18.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:18.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:18.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:17 smithi103 ceph-mon[32635]: from='osd.7 ' entity='osd.7' 2024-04-26T20:05:18.044 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_2 2024-04-26T20:05:18.291 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:05:18.667 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:18 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:05:18.667 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:18 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:05:18.667 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:18 smithi204 ceph-mon[32618]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:18.667 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:18 smithi204 ceph-mon[32618]: osd.7 [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] boot 2024-04-26T20:05:18.667 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:18 smithi204 ceph-mon[32618]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T20:05:18.667 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:18 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:18.667 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:18 smithi204 ceph-mon[32618]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T20:05:18.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:18 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:05:18.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:18 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:05:18.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:18 smithi103 ceph-mon[32635]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:18.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:18 smithi103 ceph-mon[32635]: osd.7 [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] boot 2024-04-26T20:05:18.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:18 smithi103 ceph-mon[32635]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T20:05:18.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:18 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:18.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:18 smithi103 ceph-mon[32635]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T20:05:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:18 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:05:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:18 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:05:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:18 smithi094 ceph-mon[28050]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:05:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:18 smithi094 ceph-mon[28050]: osd.7 [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] boot 2024-04-26T20:05:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:18 smithi094 ceph-mon[28050]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T20:05:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:18 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:05:18.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:18 smithi094 ceph-mon[28050]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T20:05:19.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.931+0000 7fb387209700 1 -- 172.21.15.204:0/306389545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 msgr2=0x7fb38010a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.931+0000 7fb387209700 1 --2- 172.21.15.204:0/306389545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 0x7fb38010a7c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb37800b0a0 tx=0x7fb37800b3b0 comp rx=0 tx=0).stop 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.931+0000 7fb387209700 1 -- 172.21.15.204:0/306389545 shutdown_connections 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.931+0000 7fb387209700 1 --2- 172.21.15.204:0/306389545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 0x7fb38010a7c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.931+0000 7fb387209700 1 --2- 172.21.15.204:0/306389545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb380102370 0x7fb380102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.931+0000 7fb387209700 1 --2- 172.21.15.204:0/306389545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380103570 0x7fb3801039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.931+0000 7fb387209700 1 -- 172.21.15.204:0/306389545 >> 172.21.15.204:0/306389545 conn(0x7fb3800fd910 msgr2=0x7fb3800ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.931+0000 7fb387209700 1 -- 172.21.15.204:0/306389545 shutdown_connections 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.931+0000 7fb387209700 1 -- 172.21.15.204:0/306389545 wait complete. 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb387209700 1 Processor -- start 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb387209700 1 -- start start 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb387209700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380102370 0x7fb3801984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:05:19.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb387209700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb380103570 0x7fb380198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb387209700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 0x7fb38019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb387209700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb38010deb0 con 0x7fb380103570 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb387209700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb38010dd30 con 0x7fb380102370 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb387209700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb38010e030 con 0x7fb380103f10 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb3857a6700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 0x7fb38019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb3857a6700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 0x7fb38019fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:56240/0 (socket says 172.21.15.204:56240) 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb3857a6700 1 -- 172.21.15.204:0/619420449 learned_addr learned my addr 172.21.15.204:0/619420449 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb384fa5700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380102370 0x7fb3801984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb3857a6700 1 -- 172.21.15.204:0/619420449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380102370 msgr2=0x7fb3801984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.932+0000 7fb3857a6700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380102370 0x7fb3801984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb3857a6700 1 -- 172.21.15.204:0/619420449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb380103570 msgr2=0x7fb380198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb3857a6700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb380103570 0x7fb380198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb3857a6700 1 -- 172.21.15.204:0/619420449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3801a0150 con 0x7fb380103f10 2024-04-26T20:05:19.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb384fa5700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380102370 0x7fb3801984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:05:19.937 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb3857a6700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 0x7fb38019fab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb378006070 tx=0x7fb378004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:05:19.937 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb37dffb700 1 -- 172.21.15.204:0/619420449 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378020070 con 0x7fb380103f10 2024-04-26T20:05:19.937 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb37dffb700 1 -- 172.21.15.204:0/619420449 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb378004020 con 0x7fb380103f10 2024-04-26T20:05:19.937 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb38006b990 con 0x7fb380103f10 2024-04-26T20:05:19.937 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb37dffb700 1 -- 172.21.15.204:0/619420449 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37801a700 con 0x7fb380103f10 2024-04-26T20:05:19.937 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.933+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb38006bf50 con 0x7fb380103f10 2024-04-26T20:05:19.938 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.934+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb38006a110 con 0x7fb380103f10 2024-04-26T20:05:19.938 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.935+0000 7fb37dffb700 1 -- 172.21.15.204:0/619420449 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb3780044f0 con 0x7fb380103f10 2024-04-26T20:05:19.938 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.936+0000 7fb37dffb700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb3700a3c40 0x7fb3700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:05:19.938 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.936+0000 7fb37dffb700 1 -- 172.21.15.204:0/619420449 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7fb37800f030 con 0x7fb380103f10 2024-04-26T20:05:19.938 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.936+0000 7fb384fa5700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb3700a3c40 0x7fb3700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:05:19.938 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.937+0000 7fb384fa5700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb3700a3c40 0x7fb3700a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb3801033d0 tx=0x7fb374008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:05:19.941 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:19.940+0000 7fb37dffb700 1 -- 172.21.15.204:0/619420449 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb3780041d0 con 0x7fb380103f10 2024-04-26T20:05:20.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi103 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi103 to 5696M 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:20.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: pgmap v196: 1 pgs: 1 remapped; 577 KiB data, 516 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:20.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:19 smithi204 ceph-mon[32618]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T20:05:20.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:20.151+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb38006c1b0 con 0x7fb3700a3c40 2024-04-26T20:05:20.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi103 2024-04-26T20:05:20.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi103 to 5696M 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: pgmap v196: 1 pgs: 1 remapped; 577 KiB data, 516 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:20.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:19 smithi094 ceph-mon[28050]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T20:05:20.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi103 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi103 to 5696M 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:20.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:20.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:20.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:20.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: pgmap v196: 1 pgs: 1 remapped; 577 KiB data, 516 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:20.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:19 smithi103 ceph-mon[32635]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T20:05:21.038 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:20 smithi204 ceph-mon[32618]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:05:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:05:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:20 smithi204 ceph-mon[32618]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-26T20:05:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:05:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:21.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:20 smithi204 ceph-mon[32618]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T20:05:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:20 smithi094 ceph-mon[28050]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:05:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:05:21.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:20 smithi094 ceph-mon[28050]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-26T20:05:21.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:05:21.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:21.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:20 smithi094 ceph-mon[28050]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T20:05:21.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:20 smithi103 ceph-mon[32635]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:05:21.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:05:21.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:20 smithi103 ceph-mon[32635]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-26T20:05:21.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:05:21.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:21.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:20 smithi103 ceph-mon[32635]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T20:05:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:22 smithi204 ceph-mon[32618]: pgmap v199: 1 pgs: 1 remapped; 577 KiB data, 516 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:22.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:22 smithi204 ceph-mon[32618]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T20:05:22.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:22 smithi094 ceph-mon[28050]: pgmap v199: 1 pgs: 1 remapped; 577 KiB data, 516 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:22.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:22 smithi094 ceph-mon[28050]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T20:05:23.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:22 smithi103 ceph-mon[32635]: pgmap v199: 1 pgs: 1 remapped; 577 KiB data, 516 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:23.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:22 smithi103 ceph-mon[32635]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T20:05:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.204:0/4068179167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "733fdc3e-cdc8-4f38-afd9-6bd2012145af"}]: dispatch 2024-04-26T20:05:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:23 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "733fdc3e-cdc8-4f38-afd9-6bd2012145af"}]: dispatch 2024-04-26T20:05:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:23 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "733fdc3e-cdc8-4f38-afd9-6bd2012145af"}]': finished 2024-04-26T20:05:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:23 smithi204 ceph-mon[32618]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T20:05:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:23.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.204:0/994213067' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:05:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/4068179167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "733fdc3e-cdc8-4f38-afd9-6bd2012145af"}]: dispatch 2024-04-26T20:05:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:23 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "733fdc3e-cdc8-4f38-afd9-6bd2012145af"}]: dispatch 2024-04-26T20:05:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:23 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "733fdc3e-cdc8-4f38-afd9-6bd2012145af"}]': finished 2024-04-26T20:05:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:23 smithi094 ceph-mon[28050]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T20:05:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:23.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/994213067' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:05:24.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.204:0/4068179167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "733fdc3e-cdc8-4f38-afd9-6bd2012145af"}]: dispatch 2024-04-26T20:05:24.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:23 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "733fdc3e-cdc8-4f38-afd9-6bd2012145af"}]: dispatch 2024-04-26T20:05:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:23 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "733fdc3e-cdc8-4f38-afd9-6bd2012145af"}]': finished 2024-04-26T20:05:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:23 smithi103 ceph-mon[32635]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T20:05:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:24.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.204:0/994213067' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:05:24.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:24 smithi204 ceph-mon[32618]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:24 smithi204 ceph-mon[32618]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-26T20:05:24.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:24 smithi204 ceph-mon[32618]: Cluster is now healthy 2024-04-26T20:05:24.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:24 smithi094 ceph-mon[28050]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:24.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:24 smithi094 ceph-mon[28050]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-26T20:05:24.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:24 smithi094 ceph-mon[28050]: Cluster is now healthy 2024-04-26T20:05:25.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:24 smithi103 ceph-mon[32635]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:25.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:24 smithi103 ceph-mon[32635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-26T20:05:25.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:24 smithi103 ceph-mon[32635]: Cluster is now healthy 2024-04-26T20:05:26.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:26 smithi204 ceph-mon[32618]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:26.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:26 smithi094 ceph-mon[28050]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:26 smithi103 ceph-mon[32635]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:28.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:28 smithi204 ceph-mon[32618]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:28.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:28 smithi094 ceph-mon[28050]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:29.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:28 smithi103 ceph-mon[32635]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:30.877 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:30 smithi204 ceph-mon[32618]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:30.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T20:05:30.878 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:30 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:30.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:30 smithi094 ceph-mon[28050]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:30.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T20:05:30.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:30 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:31.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:30 smithi103 ceph-mon[32635]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:31.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T20:05:31.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:30 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:31.825 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:31 smithi204 ceph-mon[32618]: Deploying daemon osd.8 on smithi204 2024-04-26T20:05:31.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:31 smithi094 ceph-mon[28050]: Deploying daemon osd.8 on smithi204 2024-04-26T20:05:32.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:31 smithi103 ceph-mon[32635]: Deploying daemon osd.8 on smithi204 2024-04-26T20:05:32.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:32 smithi204 ceph-mon[32618]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:32.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:32 smithi094 ceph-mon[28050]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:33.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:32 smithi103 ceph-mon[32635]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:34.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:34 smithi204 ceph-mon[32618]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:34.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:34 smithi094 ceph-mon[28050]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:35.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:34 smithi103 ceph-mon[32635]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:36.741 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:36 smithi204 ceph-mon[32618]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:36.741 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:36.741 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:36.741 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:36 smithi094 ceph-mon[28050]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:37.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:36 smithi103 ceph-mon[32635]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:37.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:37.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:37.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:38.857 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:38 smithi204 ceph-mon[32618]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:38.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:38 smithi094 ceph-mon[28050]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:39.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:38 smithi103 ceph-mon[32635]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:39.237 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 8 on host 'smithi204' 2024-04-26T20:05:39.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.236+0000 7fb37dffb700 1 -- 172.21.15.204:0/619420449 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb38006c1b0 con 0x7fb3700a3c40 2024-04-26T20:05:39.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.238+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb3700a3c40 msgr2=0x7fb3700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:39.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.238+0000 7fb387209700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb3700a3c40 0x7fb3700a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb3801033d0 tx=0x7fb374008040 comp rx=0 tx=0).stop 2024-04-26T20:05:39.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 msgr2=0x7fb38019fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:39.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 0x7fb38019fab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb378006070 tx=0x7fb378004ef0 comp rx=0 tx=0).stop 2024-04-26T20:05:39.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 shutdown_connections 2024-04-26T20:05:39.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb380103f10 0x7fb38019fab0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:39.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb3700a3c40 0x7fb3700a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:39.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380102370 0x7fb3801984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:39.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 --2- 172.21.15.204:0/619420449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb380103570 0x7fb380198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:39.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 >> 172.21.15.204:0/619420449 conn(0x7fb3800fd910 msgr2=0x7fb380106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:05:39.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 shutdown_connections 2024-04-26T20:05:39.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:05:39.239+0000 7fb387209700 1 -- 172.21.15.204:0/619420449 wait complete. 2024-04-26T20:05:39.873 DEBUG:teuthology.orchestra.run.smithi204:osd.8> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T20:05:39.876 INFO:tasks.cephadm:Deploying osd.9 on smithi094 with /dev/vg_nvme/lv_1... 2024-04-26T20:05:39.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T20:05:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:40 smithi094 ceph-mon[28050]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:40 smithi094 ceph-mon[28050]: from='osd.8 [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:05:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:40 smithi094 ceph-mon[28050]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:05:40.316 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:05:40.489 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.489 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.489 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.489 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.489 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:40 smithi204 ceph-mon[32618]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:40.489 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:40 smithi204 ceph-mon[32618]: from='osd.8 [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:05:40.489 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:40 smithi204 ceph-mon[32618]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:05:40.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:40.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:40 smithi103 ceph-mon[32635]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:40.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:40 smithi103 ceph-mon[32635]: from='osd.8 [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:05:40.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:40 smithi103 ceph-mon[32635]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:05:41.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:41 smithi094 ceph-mon[28050]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T20:05:41.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:41 smithi094 ceph-mon[28050]: from='osd.8 [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:05:41.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:41 smithi094 ceph-mon[28050]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T20:05:41.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:41 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:41.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:41 smithi094 ceph-mon[28050]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:05:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:41 smithi103 ceph-mon[32635]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T20:05:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:41 smithi103 ceph-mon[32635]: from='osd.8 [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:05:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:41 smithi103 ceph-mon[32635]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T20:05:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:41 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:41.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:41 smithi103 ceph-mon[32635]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:05:41.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:41 smithi204 ceph-mon[32618]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T20:05:41.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:41 smithi204 ceph-mon[32618]: from='osd.8 [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:05:41.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:41 smithi204 ceph-mon[32618]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T20:05:41.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:41 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:41.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:41 smithi204 ceph-mon[32618]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:05:42.526 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:05:42 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:05:42.269+0000 7f5adf080700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T20:05:42.527 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:05:42 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:05:42.282+0000 7f5ad666a700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:05:42.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:42 smithi094 ceph-mon[28050]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:05:42.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:42 smithi094 ceph-mon[28050]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:42.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:42 smithi094 ceph-mon[28050]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T20:05:42.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:42.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:42 smithi094 ceph-mon[28050]: from='osd.8 ' entity='osd.8' 2024-04-26T20:05:42.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:42 smithi103 ceph-mon[32635]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:05:42.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:42 smithi103 ceph-mon[32635]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:42.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:42 smithi103 ceph-mon[32635]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T20:05:42.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:42 smithi103 ceph-mon[32635]: from='osd.8 ' entity='osd.8' 2024-04-26T20:05:42.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:42 smithi204 ceph-mon[32618]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:05:42.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:42 smithi204 ceph-mon[32618]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:05:42.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:42 smithi204 ceph-mon[32618]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T20:05:42.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:42.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:42 smithi204 ceph-mon[32618]: from='osd.8 ' entity='osd.8' 2024-04-26T20:05:43.413 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T20:05:43.414 INFO:teuthology.orchestra.run.smithi094.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T20:05:43.414 INFO:teuthology.orchestra.run.smithi094.stderr: stderr: 10+0 records in 2024-04-26T20:05:43.414 INFO:teuthology.orchestra.run.smithi094.stderr:10+0 records out 2024-04-26T20:05:43.414 INFO:teuthology.orchestra.run.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0361176 s, 290 MB/s 2024-04-26T20:05:43.414 INFO:teuthology.orchestra.run.smithi094.stderr:--> Zapping successful for: 2024-04-26T20:05:43.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:05:43.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: osd.8 [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] boot 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.739 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:43.740 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:43.740 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:43.740 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.740 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:43 smithi094 ceph-mon[28050]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: osd.8 [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] boot 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:43.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:43.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:43.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:43.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:43.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:43 smithi103 ceph-mon[32635]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T20:05:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:05:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:05:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: osd.8 [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] boot 2024-04-26T20:05:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T20:05:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:05:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:43.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:05:43.884 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:05:43.884 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:43 smithi204 ceph-mon[32618]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T20:05:44.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_1 2024-04-26T20:05:44.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:44 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi204 2024-04-26T20:05:44.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:44 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi204 to 5696M 2024-04-26T20:05:44.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:44 smithi094 ceph-mon[28050]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 492 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:05:44.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:44 smithi094 ceph-mon[28050]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T20:05:44.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:44 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi204 2024-04-26T20:05:44.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:44 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi204 to 5696M 2024-04-26T20:05:44.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:44 smithi103 ceph-mon[32635]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 492 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:05:44.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:44 smithi103 ceph-mon[32635]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T20:05:44.853 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:05:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:44 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi204 2024-04-26T20:05:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:44 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi204 to 5696M 2024-04-26T20:05:44.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:44 smithi204 ceph-mon[32618]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 492 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:05:44.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:44 smithi204 ceph-mon[32618]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T20:05:46.824 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:46 smithi094 ceph-mon[28050]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 492 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:05:46.824 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:46 smithi094 ceph-mon[28050]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T20:05:46.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:46 smithi204 ceph-mon[32618]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 492 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:05:46.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:46 smithi204 ceph-mon[32618]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T20:05:47.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:46 smithi103 ceph-mon[32635]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 492 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:05:47.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:46 smithi103 ceph-mon[32635]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T20:05:47.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.114+0000 7f6617a00700 1 -- 172.21.15.94:0/4186198195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66101052c0 msgr2=0x7f6610105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:47.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.114+0000 7f6617a00700 1 --2- 172.21.15.94:0/4186198195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66101052c0 0x7f6610105720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6600009990 tx=0x7f6600009ca0 comp rx=0 tx=0).stop 2024-04-26T20:05:47.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.115+0000 7f6617a00700 1 -- 172.21.15.94:0/4186198195 shutdown_connections 2024-04-26T20:05:47.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.115+0000 7f6617a00700 1 --2- 172.21.15.94:0/4186198195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6610105c60 0x7f661010daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:47.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.115+0000 7f6617a00700 1 --2- 172.21.15.94:0/4186198195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6610106d80 0x7f66101071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:47.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.115+0000 7f6617a00700 1 --2- 172.21.15.94:0/4186198195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66101052c0 0x7f6610105720 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:47.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.115+0000 7f6617a00700 1 -- 172.21.15.94:0/4186198195 >> 172.21.15.94:0/4186198195 conn(0x7f6610100c00 msgr2=0x7f6610103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:05:47.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.115+0000 7f6617a00700 1 -- 172.21.15.94:0/4186198195 shutdown_connections 2024-04-26T20:05:47.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.115+0000 7f6617a00700 1 -- 172.21.15.94:0/4186198195 wait complete. 2024-04-26T20:05:47.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.116+0000 7f6617a00700 1 Processor -- start 2024-04-26T20:05:47.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.116+0000 7f6617a00700 1 -- start start 2024-04-26T20:05:47.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.116+0000 7f6617a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66101052c0 0x7f661019b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:05:47.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.116+0000 7f6617a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6610105c60 0x7f661019be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:05:47.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.116+0000 7f6617a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6610106d80 0x7f66101a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:05:47.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.116+0000 7f6617a00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6610111290 con 0x7f6610105c60 2024-04-26T20:05:47.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.116+0000 7f6617a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6610111110 con 0x7f66101052c0 2024-04-26T20:05:47.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.116+0000 7f6617a00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6610111410 con 0x7f6610106d80 2024-04-26T20:05:47.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66101052c0 0x7f661019b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:05:47.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6607e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6610105c60 0x7f661019be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:05:47.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66101052c0 0x7f661019b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:39794/0 (socket says 172.21.15.94:39794) 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615000700 1 -- 172.21.15.94:0/1274709766 learned_addr learned my addr 172.21.15.94:0/1274709766 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615a00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6610106d80 0x7f66101a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615000700 1 -- 172.21.15.94:0/1274709766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6610106d80 msgr2=0x7f66101a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615000700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6610106d80 0x7f66101a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615000700 1 -- 172.21.15.94:0/1274709766 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6610105c60 msgr2=0x7f661019be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615000700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6610105c60 0x7f661019be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615000700 1 -- 172.21.15.94:0/1274709766 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66101a35a0 con 0x7f66101052c0 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615a00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6610106d80 0x7f66101a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.117+0000 7f6615000700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66101052c0 0x7f661019b940 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f65f800e970 tx=0x7f65f800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.118+0000 7f660e000700 1 -- 172.21.15.94:0/1274709766 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f800cbd0 con 0x7f66101052c0 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.118+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66101a3890 con 0x7f66101052c0 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.118+0000 7f660e000700 1 -- 172.21.15.94:0/1274709766 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65f80044d0 con 0x7f66101052c0 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.118+0000 7f660e000700 1 -- 172.21.15.94:0/1274709766 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8003d60 con 0x7f66101052c0 2024-04-26T20:05:47.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.118+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66101a3e20 con 0x7f66101052c0 2024-04-26T20:05:47.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.119+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f661006a110 con 0x7f66101052c0 2024-04-26T20:05:47.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.121+0000 7f660e000700 1 -- 172.21.15.94:0/1274709766 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f65f8004000 con 0x7f66101052c0 2024-04-26T20:05:47.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.122+0000 7f660e000700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f65f00a3c40 0x7f65f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:05:47.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.123+0000 7f660e000700 1 -- 172.21.15.94:0/1274709766 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f65f80cfe10 con 0x7f66101052c0 2024-04-26T20:05:47.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.123+0000 7f6607e00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f65f00a3c40 0x7f65f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:05:47.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.123+0000 7f6607e00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f65f00a3c40 0x7f65f00a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f660000b720 tx=0x7f6600005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:05:47.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.127+0000 7f660e000700 1 -- 172.21.15.94:0/1274709766 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f65f80d3030 con 0x7f66101052c0 2024-04-26T20:05:47.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:05:47.363+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6610063860 con 0x7f65f00a3c40 2024-04-26T20:05:47.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:47 smithi204 ceph-mon[32618]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T20:05:47.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:47 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:05:47.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:47 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:05:47.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:47 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:47.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:47 smithi094 ceph-mon[28050]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T20:05:47.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:47 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:05:47.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:47 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:05:47.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:47 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:48.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:47 smithi103 ceph-mon[32635]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T20:05:48.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:47 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:05:48.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:47 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:05:48.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:47 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:05:48.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:48 smithi204 ceph-mon[32618]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:05:48.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:48 smithi204 ceph-mon[32618]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 492 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:05:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:48 smithi094 ceph-mon[28050]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:05:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:48 smithi094 ceph-mon[28050]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 492 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:05:49.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:48 smithi103 ceph-mon[32635]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:05:49.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:48 smithi103 ceph-mon[32635]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 492 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:05:50.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:50 smithi204 ceph-mon[32618]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:05:50.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:50 smithi094 ceph-mon[28050]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:05:51.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:50 smithi103 ceph-mon[32635]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:05:51.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:51 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3601259178' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bdd2b36-7b98-470b-b0c1-659406d966b7"}]: dispatch 2024-04-26T20:05:51.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:51 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3601259178' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bdd2b36-7b98-470b-b0c1-659406d966b7"}]': finished 2024-04-26T20:05:51.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:51 smithi204 ceph-mon[32618]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T20:05:51.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:51 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:05:51.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:51 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3601259178' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bdd2b36-7b98-470b-b0c1-659406d966b7"}]: dispatch 2024-04-26T20:05:51.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:51 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3601259178' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bdd2b36-7b98-470b-b0c1-659406d966b7"}]': finished 2024-04-26T20:05:51.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:51 smithi094 ceph-mon[28050]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T20:05:51.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:51 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:05:52.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:51 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3601259178' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bdd2b36-7b98-470b-b0c1-659406d966b7"}]: dispatch 2024-04-26T20:05:52.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:51 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3601259178' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bdd2b36-7b98-470b-b0c1-659406d966b7"}]': finished 2024-04-26T20:05:52.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:51 smithi103 ceph-mon[32635]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T20:05:52.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:51 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:05:52.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:52 smithi204 ceph-mon[32618]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:05:52.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:52 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3931913655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:05:52.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:52 smithi094 ceph-mon[28050]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:05:52.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:52 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3931913655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:05:53.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:52 smithi103 ceph-mon[32635]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:05:53.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:52 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3931913655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:05:54.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:54 smithi103 ceph-mon[32635]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:05:54.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:54 smithi204 ceph-mon[32618]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:05:54.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:54 smithi094 ceph-mon[28050]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:05:56.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:56 smithi204 ceph-mon[32618]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T20:05:56.986 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:56 smithi094 ceph-mon[28050]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T20:05:57.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:56 smithi103 ceph-mon[32635]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T20:05:58.860 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:05:58 smithi094 ceph-mon[28050]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:05:58.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:05:58 smithi204 ceph-mon[32618]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:05:59.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:05:58 smithi103 ceph-mon[32635]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:06:00.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:00 smithi204 ceph-mon[32618]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:00.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:00 smithi094 ceph-mon[28050]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:01.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:00 smithi103 ceph-mon[32635]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:01.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:01 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T20:06:01.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:01 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:01.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T20:06:01.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:02.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:01 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T20:06:02.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:01 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:02.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:02 smithi094 ceph-mon[28050]: Deploying daemon osd.9 on smithi094 2024-04-26T20:06:02.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:02 smithi094 ceph-mon[28050]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:02.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:02 smithi204 ceph-mon[32618]: Deploying daemon osd.9 on smithi094 2024-04-26T20:06:02.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:02 smithi204 ceph-mon[32618]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:03.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:02 smithi103 ceph-mon[32635]: Deploying daemon osd.9 on smithi094 2024-04-26T20:06:03.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:02 smithi103 ceph-mon[32635]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:04.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:04 smithi204 ceph-mon[32618]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:04.907 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:04 smithi094 ceph-mon[28050]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:05.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:04 smithi103 ceph-mon[32635]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:06.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:06 smithi204 ceph-mon[32618]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:06.904 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:06 smithi094 ceph-mon[28050]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:07.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:06 smithi103 ceph-mon[32635]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:08.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:08 smithi204 ceph-mon[32618]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:08.906 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:08 smithi094 ceph-mon[28050]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:09.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:08 smithi103 ceph-mon[32635]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:10.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:10 smithi204 ceph-mon[32618]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:10 smithi094 ceph-mon[28050]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:10.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:10 smithi103 ceph-mon[32635]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:12.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:12 smithi204 ceph-mon[32618]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:12.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:12 smithi094 ceph-mon[28050]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:13.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:12 smithi103 ceph-mon[32635]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:14.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:14 smithi204 ceph-mon[32618]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:14.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:14.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:14.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:14 smithi204 ceph-mon[32618]: from='osd.9 [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:06:14.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:14 smithi204 ceph-mon[32618]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:06:14.936 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:14 smithi094 ceph-mon[28050]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:14.983 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:14.990 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:14.997 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:14 smithi094 ceph-mon[28050]: from='osd.9 [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:06:15.004 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:14 smithi094 ceph-mon[28050]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:06:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.011+0000 7f660e000700 1 -- 172.21.15.94:0/1274709766 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6610063860 con 0x7f65f00a3c40 2024-04-26T20:06:15.021 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 9 on host 'smithi094' 2024-04-26T20:06:15.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.014+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f65f00a3c40 msgr2=0x7f65f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:15.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.014+0000 7f6617a00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f65f00a3c40 0x7f65f00a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f660000b720 tx=0x7f6600005fd0 comp rx=0 tx=0).stop 2024-04-26T20:06:15.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.014+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66101052c0 msgr2=0x7f661019b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:15.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.014+0000 7f6617a00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66101052c0 0x7f661019b940 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f65f800e970 tx=0x7f65f800ec80 comp rx=0 tx=0).stop 2024-04-26T20:06:15.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.015+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 shutdown_connections 2024-04-26T20:06:15.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.015+0000 7f6617a00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6610106d80 0x7f66101a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:15.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.015+0000 7f6617a00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f65f00a3c40 0x7f65f00a6100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.015+0000 7f6617a00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66101052c0 0x7f661019b940 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:15.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.015+0000 7f6617a00700 1 --2- 172.21.15.94:0/1274709766 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6610105c60 0x7f661019be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:15.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.015+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 >> 172.21.15.94:0/1274709766 conn(0x7f6610100c00 msgr2=0x7f6610109c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:06:15.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.015+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 shutdown_connections 2024-04-26T20:06:15.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:06:15.015+0000 7f6617a00700 1 -- 172.21.15.94:0/1274709766 wait complete. 2024-04-26T20:06:15.280 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:14 smithi103 ceph-mon[32635]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:15.291 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:15.302 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:15.311 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:14 smithi103 ceph-mon[32635]: from='osd.9 [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:06:15.319 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:14 smithi103 ceph-mon[32635]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:06:15.827 DEBUG:teuthology.orchestra.run.smithi094:osd.9> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T20:06:15.852 INFO:tasks.cephadm:Deploying osd.10 on smithi103 with /dev/vg_nvme/lv_1... 2024-04-26T20:06:15.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T20:06:15.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:15 smithi094 ceph-mon[28050]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T20:06:16.002 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:15 smithi094 ceph-mon[28050]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T20:06:16.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:16.084 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:15 smithi094 ceph-mon[28050]: from='osd.9 [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:06:16.166 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:15 smithi094 ceph-mon[28050]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:06:16.176 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:16.184 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:16.193 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:15 smithi094 ceph-mon[28050]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:16.203 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:15 smithi204 ceph-mon[32618]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T20:06:16.218 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:15 smithi204 ceph-mon[32618]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T20:06:16.228 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:16.235 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:15 smithi204 ceph-mon[32618]: from='osd.9 [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:06:16.244 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:15 smithi204 ceph-mon[32618]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:06:16.252 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:16.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:16.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:15 smithi204 ceph-mon[32618]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:16.328 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:15 smithi103 ceph-mon[32635]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T20:06:16.340 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:15 smithi103 ceph-mon[32635]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T20:06:16.349 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:16.357 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:15 smithi103 ceph-mon[32635]: from='osd.9 [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:06:16.475 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:15 smithi103 ceph-mon[32635]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]: dispatch 2024-04-26T20:06:16.485 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:16.493 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:16.502 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:15 smithi103 ceph-mon[32635]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:16.540 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:06:16.723 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:06:16 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:06:16.454+0000 7fa927e00700 -1 osd.9 0 waiting for initial osdmap 2024-04-26T20:06:16.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:06:16 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:06:16.477+0000 7fa91dc00700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:06:16.955 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:16 smithi103 ceph-mon[32635]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:06:17.042 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:16 smithi103 ceph-mon[32635]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T20:06:17.061 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:16 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:17.070 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:16 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:17.078 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:16 smithi103 ceph-mon[32635]: from='osd.9 ' entity='osd.9' 2024-04-26T20:06:17.086 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:16 smithi094 ceph-mon[28050]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:06:17.093 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:16 smithi094 ceph-mon[28050]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T20:06:17.106 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:16 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:17.115 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:16 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:17.122 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:16 smithi094 ceph-mon[28050]: from='osd.9 ' entity='osd.9' 2024-04-26T20:06:17.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:16 smithi204 ceph-mon[32618]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-04-26T20:06:17.194 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:16 smithi204 ceph-mon[32618]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T20:06:17.200 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:16 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:17.208 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:16 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:17.215 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:16 smithi204 ceph-mon[32618]: from='osd.9 ' entity='osd.9' 2024-04-26T20:06:17.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:17 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:06:18.021 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:17 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:06:18.048 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:18.057 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:17 smithi094 ceph-mon[28050]: osd.9 [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] boot 2024-04-26T20:06:18.068 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:17 smithi094 ceph-mon[28050]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T20:06:18.077 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:18.085 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:17 smithi094 ceph-mon[28050]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:18.094 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:17 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:06:18.103 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:17 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:06:18.115 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:18.130 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:17 smithi103 ceph-mon[32635]: osd.9 [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] boot 2024-04-26T20:06:18.137 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:17 smithi103 ceph-mon[32635]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T20:06:18.240 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:18.305 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:17 smithi103 ceph-mon[32635]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:18.320 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:17 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:06:18.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:17 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:06:18.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:18.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:17 smithi204 ceph-mon[32618]: osd.9 [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] boot 2024-04-26T20:06:18.359 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:17 smithi204 ceph-mon[32618]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T20:06:18.418 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:06:18.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:17 smithi204 ceph-mon[32618]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:06:18.434 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T20:06:18.443 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T20:06:18.452 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-26T20:06:18.460 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-26T20:06:18.468 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114497 s, 916 MB/s 2024-04-26T20:06:18.477 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-26T20:06:18.950 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_1 2024-04-26T20:06:19.242 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:06:19.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:19 smithi094 ceph-mon[28050]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T20:06:20.008 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:19 smithi094 ceph-mon[28050]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:20.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:19 smithi103 ceph-mon[32635]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T20:06:20.043 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:19 smithi103 ceph-mon[32635]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:20.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:19 smithi204 ceph-mon[32618]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T20:06:20.139 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:19 smithi204 ceph-mon[32618]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:20.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.871+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1064910956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed01035a0 msgr2=0x7f0ed0103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:20.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.871+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1064910956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed01035a0 0x7f0ed0103a00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0099c0 tx=0x7f0ebc009cd0 comp rx=0 tx=0).stop 2024-04-26T20:06:20.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.871+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1064910956 shutdown_connections 2024-04-26T20:06:20.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.871+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1064910956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ed0103f40 0x7f0ed010a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:20.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.871+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1064910956 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed01023a0 0x7f0ed01027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:20.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.871+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1064910956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed01035a0 0x7f0ed0103a00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:20.951 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.871+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1064910956 >> 172.21.15.103:0/1064910956 conn(0x7f0ed00fd980 msgr2=0x7f0ed00ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:06:21.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1064910956 shutdown_connections 2024-04-26T20:06:21.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1064910956 wait complete. 2024-04-26T20:06:21.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 Processor -- start 2024-04-26T20:06:21.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 -- start start 2024-04-26T20:06:21.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed01023a0 0x7f0ed0199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:06:21.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ed01035a0 0x7f0ed0199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:06:21.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed0103f40 0x7f0ed01a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:06:21.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed010ded0 con 0x7f0ed01023a0 2024-04-26T20:06:21.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed010dd50 con 0x7f0ed0103f40 2024-04-26T20:06:21.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed6e80700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed010e050 con 0x7f0ed01035a0 2024-04-26T20:06:21.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.872+0000 7f0ed541d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed0103f40 0x7f0ed01a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:06:21.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed541d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed0103f40 0x7f0ed01a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43542/0 (socket says 172.21.15.103:43542) 2024-04-26T20:06:21.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed541d700 1 -- 172.21.15.103:0/1467420729 learned_addr learned my addr 172.21.15.103:0/1467420729 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T20:06:21.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ecffff700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ed01035a0 0x7f0ed0199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:06:21.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed4c1c700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed01023a0 0x7f0ed0199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:06:21.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed541d700 1 -- 172.21.15.103:0/1467420729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ed01035a0 msgr2=0x7f0ed0199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:21.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed541d700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ed01035a0 0x7f0ed0199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:21.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed541d700 1 -- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed01023a0 msgr2=0x7f0ed0199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:21.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed541d700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed01023a0 0x7f0ed0199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:21.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed541d700 1 -- 172.21.15.103:0/1467420729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed01a12c0 con 0x7f0ed0103f40 2024-04-26T20:06:21.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed541d700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed0103f40 0x7f0ed01a0c20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800ada0 tx=0x7f0ec800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:06:21.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ecdffb700 1 -- 172.21.15.103:0/1467420729 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8017020 con 0x7f0ed0103f40 2024-04-26T20:06:21.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ecdffb700 1 -- 172.21.15.103:0/1467420729 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ec800ce30 con 0x7f0ed0103f40 2024-04-26T20:06:21.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed01a15b0 con 0x7f0ed0103f40 2024-04-26T20:06:21.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ecdffb700 1 -- 172.21.15.103:0/1467420729 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec80053a0 con 0x7f0ed0103f40 2024-04-26T20:06:21.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.873+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed01a1af0 con 0x7f0ed0103f40 2024-04-26T20:06:21.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.874+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ed006a110 con 0x7f0ed0103f40 2024-04-26T20:06:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.875+0000 7f0ecdffb700 1 -- 172.21.15.103:0/1467420729 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0ec800c980 con 0x7f0ed0103f40 2024-04-26T20:06:21.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.876+0000 7f0ecdffb700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0ec00a3c40 0x7f0ec00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:06:21.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.876+0000 7f0ecdffb700 1 -- 172.21.15.103:0/1467420729 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f0ec80cf2b0 con 0x7f0ed0103f40 2024-04-26T20:06:21.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.876+0000 7f0ed4c1c700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0ec00a3c40 0x7f0ec00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:06:21.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.877+0000 7f0ed4c1c700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0ec00a3c40 0x7f0ec00a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4006fd0 tx=0x7f0ec4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:06:21.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:20.880+0000 7f0ecdffb700 1 -- 172.21.15.103:0/1467420729 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0ec8021020 con 0x7f0ed0103f40 2024-04-26T20:06:21.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:21.099+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0ed006c240 con 0x7f0ec00a3c40 2024-04-26T20:06:21.972 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi094 2024-04-26T20:06:21.998 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.140 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.149 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.202 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi094 to 4272M 2024-04-26T20:06:22.221 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.228 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:22.245 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:22.275 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.288 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:22.350 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:22.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:22.376 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.389 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:06:22.396 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:06:22.409 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:06:22.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:22.433 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:21 smithi103 ceph-mon[32635]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:22.492 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi094 2024-04-26T20:06:22.515 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.521 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.534 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.541 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.556 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.563 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi094 to 4272M 2024-04-26T20:06:22.585 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.625 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:22.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:22.647 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.654 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:22.669 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:22.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:22.693 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.700 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:06:22.753 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:06:22.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:06:22.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:22.832 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:21 smithi204 ceph-mon[32618]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:22.846 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi094 2024-04-26T20:06:22.855 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.870 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.880 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.896 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:22.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi094 to 4272M 2024-04-26T20:06:22.972 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:22.981 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:22.998 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:23.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:23.024 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:23.099 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:23.177 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:23.186 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:23.200 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:06:23.209 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:06:23.229 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:06:23.236 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:23.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:21 smithi094 ceph-mon[28050]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:24.835 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:24 smithi204 ceph-mon[32618]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:24.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:24 smithi204 ceph-mon[32618]: from='client.? 172.21.15.103:0/1385990416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2"}]: dispatch 2024-04-26T20:06:24.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:24 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2"}]: dispatch 2024-04-26T20:06:24.910 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:24 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2"}]': finished 2024-04-26T20:06:24.919 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:24 smithi204 ceph-mon[32618]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T20:06:24.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:24 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:24.939 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:24 smithi204 ceph-mon[32618]: from='client.? 172.21.15.103:0/102443829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:06:24.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:24 smithi094 ceph-mon[28050]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:24.995 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:24 smithi094 ceph-mon[28050]: from='client.? 172.21.15.103:0/1385990416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2"}]: dispatch 2024-04-26T20:06:25.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:24 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2"}]: dispatch 2024-04-26T20:06:25.127 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:24 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2"}]': finished 2024-04-26T20:06:25.138 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:24 smithi094 ceph-mon[28050]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T20:06:25.149 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:24 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:25.158 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:24 smithi094 ceph-mon[28050]: from='client.? 172.21.15.103:0/102443829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:06:25.168 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:24 smithi103 ceph-mon[32635]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:25.179 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:24 smithi103 ceph-mon[32635]: from='client.? 172.21.15.103:0/1385990416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2"}]: dispatch 2024-04-26T20:06:25.189 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:24 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2"}]: dispatch 2024-04-26T20:06:25.197 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:24 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2"}]': finished 2024-04-26T20:06:25.253 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:24 smithi103 ceph-mon[32635]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T20:06:25.330 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:24 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:25.336 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:24 smithi103 ceph-mon[32635]: from='client.? 172.21.15.103:0/102443829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:06:26.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:26 smithi204 ceph-mon[32618]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:26.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:26 smithi094 ceph-mon[28050]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:26 smithi103 ceph-mon[32635]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:28.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:28 smithi204 ceph-mon[32618]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:28.986 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:28 smithi103 ceph-mon[32635]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:29.000 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:28 smithi094 ceph-mon[28050]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:30 smithi204 ceph-mon[32618]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:30.953 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:30 smithi103 ceph-mon[32635]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:30.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:30 smithi094 ceph-mon[28050]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:31.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:31 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T20:06:31.976 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:31 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:31.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T20:06:31.995 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:32.001 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:31 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T20:06:32.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:31 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:32.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:32 smithi204 ceph-mon[32618]: Deploying daemon osd.10 on smithi103 2024-04-26T20:06:32.927 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:32 smithi204 ceph-mon[32618]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:32.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:32 smithi094 ceph-mon[28050]: Deploying daemon osd.10 on smithi103 2024-04-26T20:06:33.020 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:32 smithi094 ceph-mon[28050]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:33.073 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:32 smithi103 ceph-mon[32635]: Deploying daemon osd.10 on smithi103 2024-04-26T20:06:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:32 smithi103 ceph-mon[32635]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:34.736 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:34 smithi103 ceph-mon[32635]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:34.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:34 smithi204 ceph-mon[32618]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:34.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:34 smithi094 ceph-mon[28050]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:36.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:36 smithi103 ceph-mon[32635]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:36.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:36 smithi204 ceph-mon[32618]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:36 smithi094 ceph-mon[28050]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:38.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:38 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:38.542 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:38 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:38.549 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:38 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:38.557 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:38 smithi094 ceph-mon[28050]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:38.565 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:38 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:38.573 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:38 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:38.582 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:38 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:38.589 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:38 smithi103 ceph-mon[32635]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:38.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:38 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:38.763 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:38 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:38.771 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:38 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:38.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:38 smithi204 ceph-mon[32618]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:40.752 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:40 smithi103 ceph-mon[32635]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:40.834 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:40.842 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:40.851 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:40.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:40.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:40 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:40.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:40 smithi204 ceph-mon[32618]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:40.923 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:40.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:40.940 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:40.948 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:40.957 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:40 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:40.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:40 smithi094 ceph-mon[28050]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:40.996 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:41.005 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:41.014 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:41.023 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:41.031 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:40 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:41.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.130+0000 7f0ecdffb700 1 -- 172.21.15.103:0/1467420729 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f0ed006c240 con 0x7f0ec00a3c40 2024-04-26T20:06:41.157 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 10 on host 'smithi103' 2024-04-26T20:06:41.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0ec00a3c40 msgr2=0x7f0ec00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0ec00a3c40 0x7f0ec00a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4006fd0 tx=0x7f0ec4008040 comp rx=0 tx=0).stop 2024-04-26T20:06:41.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed0103f40 msgr2=0x7f0ed01a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:41.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed0103f40 0x7f0ed01a0c20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800ada0 tx=0x7f0ec800c330 comp rx=0 tx=0).stop 2024-04-26T20:06:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 shutdown_connections 2024-04-26T20:06:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f0ec00a3c40 0x7f0ec00a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ed0103f40 0x7f0ed01a0c20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:41.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed01023a0 0x7f0ed0199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 --2- 172.21.15.103:0/1467420729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ed01035a0 0x7f0ed0199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 >> 172.21.15.103:0/1467420729 conn(0x7f0ed00fd980 msgr2=0x7f0ed006bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:06:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 shutdown_connections 2024-04-26T20:06:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T20:06:41.133+0000 7f0ed6e80700 1 -- 172.21.15.103:0/1467420729 wait complete. 2024-04-26T20:06:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:41 smithi103 ceph-mon[32635]: from='osd.10 [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:06:41.793 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:41 smithi103 ceph-mon[32635]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:06:41.801 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:41 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:41.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:41 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:41.870 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:41 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:41.880 DEBUG:teuthology.orchestra.run.smithi103:osd.10> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T20:06:41.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:41 smithi204 ceph-mon[32618]: from='osd.10 [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:06:41.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:41 smithi204 ceph-mon[32618]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:06:41.938 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:41 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:41.947 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:41 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:41.955 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:41 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:41.965 INFO:tasks.cephadm:Deploying osd.11 on smithi204 with /dev/vg_nvme/lv_1... 2024-04-26T20:06:41.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T20:06:41.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:41 smithi094 ceph-mon[28050]: from='osd.10 [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:06:41.995 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:41 smithi094 ceph-mon[28050]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:06:42.004 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:41 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:42.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:41 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:06:42.020 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:41 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:42.230 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:06:42.749 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:42 smithi103 ceph-mon[32635]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:42.783 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:42 smithi103 ceph-mon[32635]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T20:06:42.790 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:42 smithi103 ceph-mon[32635]: from='osd.10 [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:06:42.797 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:42 smithi103 ceph-mon[32635]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T20:06:42.805 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:42.813 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:42 smithi103 ceph-mon[32635]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:06:42.869 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:06:42 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:06:42.598+0000 7f9de90c1700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T20:06:42.878 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:06:42 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:06:42.610+0000 7f9de4eb4700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:06:42.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:42 smithi204 ceph-mon[32618]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:42.895 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:42 smithi204 ceph-mon[32618]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T20:06:42.904 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:42 smithi204 ceph-mon[32618]: from='osd.10 [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:06:42.913 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:42 smithi204 ceph-mon[32618]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T20:06:42.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:42 smithi204 ceph-mon[32618]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:06:42.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:42 smithi094 ceph-mon[28050]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:42.995 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:42 smithi094 ceph-mon[28050]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T20:06:43.003 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:42 smithi094 ceph-mon[28050]: from='osd.10 [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:06:43.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:42 smithi094 ceph-mon[28050]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T20:06:43.084 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:43.093 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:42 smithi094 ceph-mon[28050]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T20:06:43.880 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:43 smithi204 ceph-mon[32618]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:06:43.894 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:43 smithi204 ceph-mon[32618]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T20:06:43.903 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:43 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:43.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:43 smithi094 ceph-mon[28050]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:06:44.002 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:43 smithi094 ceph-mon[28050]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T20:06:44.011 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:43 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:44.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:43 smithi103 ceph-mon[32635]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T20:06:44.043 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:43 smithi103 ceph-mon[32635]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T20:06:44.051 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:43 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:44.274 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T20:06:44.292 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T20:06:44.300 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-04-26T20:06:44.309 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-04-26T20:06:44.317 INFO:teuthology.orchestra.run.smithi204.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130645 s, 803 MB/s 2024-04-26T20:06:44.325 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 2024-04-26T20:06:44.413 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-04-26T20:06:44.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:44 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:06:44.793 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:44 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:06:44.800 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:44 smithi103 ceph-mon[32635]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:44.810 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:44 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:44.864 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:44 smithi103 ceph-mon[32635]: osd.10 [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] boot 2024-04-26T20:06:44.872 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:44 smithi103 ceph-mon[32635]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T20:06:44.881 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:44 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:44.889 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:44 smithi103 ceph-mon[32635]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T20:06:44.899 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:44 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:06:44.908 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:44 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:06:44.916 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:44 smithi094 ceph-mon[28050]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:44.958 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:44 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:44.966 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:44 smithi094 ceph-mon[28050]: osd.10 [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] boot 2024-04-26T20:06:44.975 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:44 smithi094 ceph-mon[28050]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T20:06:44.984 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:44 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:45.013 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:44 smithi094 ceph-mon[28050]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T20:06:45.080 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:44 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:06:45.109 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:44 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:06:45.117 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:44 smithi204 ceph-mon[32618]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:06:45.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:44 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:45.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:44 smithi204 ceph-mon[32618]: osd.10 [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] boot 2024-04-26T20:06:45.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:44 smithi204 ceph-mon[32618]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T20:06:45.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:44 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:06:45.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:44 smithi204 ceph-mon[32618]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T20:06:45.197 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_1 2024-04-26T20:06:45.443 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:06:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi103 2024-04-26T20:06:46.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.161 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.170 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.193 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.263 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi103 to 4272M 2024-04-26T20:06:46.271 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.278 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:46.285 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:46.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T20:06:46.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:45 smithi204 ceph-mon[32618]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:06:46.364 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi103 2024-04-26T20:06:46.373 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.382 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.390 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.435 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.474 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.496 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi103 to 4272M 2024-04-26T20:06:46.504 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.512 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:46.561 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:46.611 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T20:06:46.629 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:45 smithi103 ceph-mon[32635]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:06:46.637 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi103 2024-04-26T20:06:46.671 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.678 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.685 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.694 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.703 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.709 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:06:46.754 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi103 to 4272M 2024-04-26T20:06:46.778 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:46.792 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:06:46.799 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:06:46.805 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T20:06:46.860 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:45 smithi094 ceph-mon[28050]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:06:47.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.036+0000 7fe957d21700 1 -- 172.21.15.204:0/450612433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe950077450 msgr2=0x7fe9500758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:47.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.036+0000 7fe957d21700 1 --2- 172.21.15.204:0/450612433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe950077450 0x7fe9500758b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00b0a0 tx=0x7fe94c00b3b0 comp rx=0 tx=0).stop 2024-04-26T20:06:47.055 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.036+0000 7fe957d21700 1 -- 172.21.15.204:0/450612433 shutdown_connections 2024-04-26T20:06:47.064 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.036+0000 7fe957d21700 1 --2- 172.21.15.204:0/450612433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe95010b380 0x7fe95010d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:47.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.036+0000 7fe957d21700 1 --2- 172.21.15.204:0/450612433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe950077450 0x7fe9500758b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:47.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.036+0000 7fe957d21700 1 --2- 172.21.15.204:0/450612433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe950075e80 0x7fe950076300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:47.122 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.036+0000 7fe957d21700 1 -- 172.21.15.204:0/450612433 >> 172.21.15.204:0/450612433 conn(0x7fe9500fea70 msgr2=0x7fe950100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:06:47.130 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.036+0000 7fe957d21700 1 -- 172.21.15.204:0/450612433 shutdown_connections 2024-04-26T20:06:47.138 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.036+0000 7fe957d21700 1 -- 172.21.15.204:0/450612433 wait complete. 2024-04-26T20:06:47.147 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe957d21700 1 Processor -- start 2024-04-26T20:06:47.192 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe957d21700 1 -- start start 2024-04-26T20:06:47.201 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe957d21700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950075e80 0x7fe95019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:06:47.209 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe957d21700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe950077450 0x7fe95019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:06:47.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe957d21700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe95010b380 0x7fe9501a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:06:47.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe957d21700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe9501111c0 con 0x7fe950077450 2024-04-26T20:06:47.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe957d21700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe950111040 con 0x7fe950075e80 2024-04-26T20:06:47.290 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe957d21700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe950111340 con 0x7fe95010b380 2024-04-26T20:06:47.333 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe9562be700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe95010b380 0x7fe9501a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:06:47.342 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe9562be700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe95010b380 0x7fe9501a2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:35500/0 (socket says 172.21.15.204:35500) 2024-04-26T20:06:47.351 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe9562be700 1 -- 172.21.15.204:0/3466564482 learned_addr learned my addr 172.21.15.204:0/3466564482 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:06:47.359 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe955abd700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950075e80 0x7fe95019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:06:47.396 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe9562be700 1 -- 172.21.15.204:0/3466564482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950075e80 msgr2=0x7fe95019b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:47.405 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe9562be700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950075e80 0x7fe95019b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:47.422 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe9562be700 1 -- 172.21.15.204:0/3466564482 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe950077450 msgr2=0x7fe95019bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:06:47.431 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe9562be700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe950077450 0x7fe95019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:06:47.440 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.037+0000 7fe9562be700 1 -- 172.21.15.204:0/3466564482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9501a3450 con 0x7fe95010b380 2024-04-26T20:06:47.448 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.038+0000 7fe9562be700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe95010b380 0x7fe9501a2db0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe94800c930 tx=0x7fe94800ccf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:06:47.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.038+0000 7fe946ffd700 1 -- 172.21.15.204:0/3466564482 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe948007ac0 con 0x7fe95010b380 2024-04-26T20:06:47.550 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.038+0000 7fe946ffd700 1 -- 172.21.15.204:0/3466564482 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe948007c60 con 0x7fe95010b380 2024-04-26T20:06:47.559 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.038+0000 7fe946ffd700 1 -- 172.21.15.204:0/3466564482 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe948018660 con 0x7fe95010b380 2024-04-26T20:06:47.567 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.038+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9501a3740 con 0x7fe95010b380 2024-04-26T20:06:47.575 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.038+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9501a3d00 con 0x7fe95010b380 2024-04-26T20:06:47.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.039+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe95006a110 con 0x7fe95010b380 2024-04-26T20:06:47.653 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.040+0000 7fe946ffd700 1 -- 172.21.15.204:0/3466564482 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe94800f450 con 0x7fe95010b380 2024-04-26T20:06:47.661 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.041+0000 7fe946ffd700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe93c0a3cc0 0x7fe93c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:06:47.674 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.041+0000 7fe946ffd700 1 -- 172.21.15.204:0/3466564482 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7fe9480cfe50 con 0x7fe95010b380 2024-04-26T20:06:47.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.041+0000 7fe955abd700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe93c0a3cc0 0x7fe93c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:06:47.693 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.042+0000 7fe955abd700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe93c0a3cc0 0x7fe93c0a6180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe94c0062a0 tx=0x7fe94c0061f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:06:47.702 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.044+0000 7fe946ffd700 1 -- 172.21.15.204:0/3466564482 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe948014030 con 0x7fe95010b380 2024-04-26T20:06:47.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:06:47.253+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe950063860 con 0x7fe93c0a3cc0 2024-04-26T20:06:48.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:47 smithi204 ceph-mon[32618]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T20:06:48.140 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:47 smithi204 ceph-mon[32618]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:06:48.154 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:47 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:06:48.165 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:47 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:06:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:47 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:48.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:47 smithi204 ceph-mon[32618]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:06:48.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:47 smithi094 ceph-mon[28050]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T20:06:48.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:47 smithi094 ceph-mon[28050]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:06:48.268 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:47 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:06:48.281 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:47 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:06:48.362 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:47 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:48.394 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:47 smithi094 ceph-mon[28050]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:06:48.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:47 smithi103 ceph-mon[32635]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T20:06:48.415 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:47 smithi103 ceph-mon[32635]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:06:48.428 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:47 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:06:48.471 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:47 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:06:48.480 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:47 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:48.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:47 smithi103 ceph-mon[32635]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:06:48.995 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:48 smithi204 ceph-mon[32618]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T20:06:49.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:48 smithi094 ceph-mon[28050]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T20:06:49.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:48 smithi103 ceph-mon[32635]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T20:06:50.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:49 smithi204 ceph-mon[32618]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-26T20:06:50.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:49 smithi204 ceph-mon[32618]: from='client.? 172.21.15.204:0/3330605942' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc362076-e2fb-4994-97c6-08e2cee8e0e0"}]: dispatch 2024-04-26T20:06:50.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:49 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc362076-e2fb-4994-97c6-08e2cee8e0e0"}]: dispatch 2024-04-26T20:06:50.169 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:49 smithi204 ceph-mon[32618]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc362076-e2fb-4994-97c6-08e2cee8e0e0"}]': finished 2024-04-26T20:06:50.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:49 smithi204 ceph-mon[32618]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T20:06:50.193 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:49 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:06:50.318 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:49 smithi103 ceph-mon[32635]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-26T20:06:50.325 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:49 smithi103 ceph-mon[32635]: from='client.? 172.21.15.204:0/3330605942' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc362076-e2fb-4994-97c6-08e2cee8e0e0"}]: dispatch 2024-04-26T20:06:50.335 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:49 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc362076-e2fb-4994-97c6-08e2cee8e0e0"}]: dispatch 2024-04-26T20:06:50.344 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:49 smithi103 ceph-mon[32635]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc362076-e2fb-4994-97c6-08e2cee8e0e0"}]': finished 2024-04-26T20:06:50.352 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:49 smithi103 ceph-mon[32635]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T20:06:50.375 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:49 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:06:50.383 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:49 smithi094 ceph-mon[28050]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-26T20:06:50.392 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:49 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/3330605942' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc362076-e2fb-4994-97c6-08e2cee8e0e0"}]: dispatch 2024-04-26T20:06:50.402 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:49 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc362076-e2fb-4994-97c6-08e2cee8e0e0"}]: dispatch 2024-04-26T20:06:50.410 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:49 smithi094 ceph-mon[28050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc362076-e2fb-4994-97c6-08e2cee8e0e0"}]': finished 2024-04-26T20:06:50.417 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:49 smithi094 ceph-mon[28050]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T20:06:50.427 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:49 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:06:51.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:50 smithi204 ceph-mon[32618]: from='client.? 172.21.15.204:0/2695997209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:06:51.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:50 smithi094 ceph-mon[28050]: from='client.? 172.21.15.204:0/2695997209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:06:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:50 smithi103 ceph-mon[32635]: from='client.? 172.21.15.204:0/2695997209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:06:52.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:51 smithi204 ceph-mon[32618]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:06:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:51 smithi094 ceph-mon[28050]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:06:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:51 smithi103 ceph-mon[32635]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:06:54.723 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:54 smithi204 ceph-mon[32618]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-26T20:06:54.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:54 smithi094 ceph-mon[28050]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-26T20:06:55.041 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:54 smithi103 ceph-mon[32635]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-26T20:06:56.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:56 smithi204 ceph-mon[32618]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:06:56.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:56 smithi094 ceph-mon[28050]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:06:57.046 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:56 smithi103 ceph-mon[32635]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:06:57.849 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T20:06:57.861 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:57.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T20:06:57.993 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:58.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T20:06:58.042 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:06:58.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:58 smithi204 ceph-mon[32618]: Deploying daemon osd.11 on smithi204 2024-04-26T20:06:58.976 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:06:58 smithi204 ceph-mon[32618]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:06:58.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:58 smithi094 ceph-mon[28050]: Deploying daemon osd.11 on smithi204 2024-04-26T20:06:58.996 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:06:58 smithi094 ceph-mon[28050]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:06:59.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:58 smithi103 ceph-mon[32635]: Deploying daemon osd.11 on smithi204 2024-04-26T20:06:59.042 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:06:58 smithi103 ceph-mon[32635]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:00.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:00 smithi094 ceph-mon[28050]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:00.771 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:00 smithi204 ceph-mon[32618]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:00.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:00 smithi103 ceph-mon[32635]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:02.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:02 smithi204 ceph-mon[32618]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:02.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:02 smithi094 ceph-mon[28050]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:03.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:02 smithi103 ceph-mon[32635]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:03.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:03.941 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:03.950 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:03.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:03.995 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:04.003 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:04.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:04.068 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:04.075 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:04.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:04 smithi204 ceph-mon[32618]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:04.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:04 smithi094 ceph-mon[28050]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:05.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:04 smithi103 ceph-mon[32635]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:06.338 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 11 on host 'smithi204' 2024-04-26T20:07:06.366 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.337+0000 7fe946ffd700 1 -- 172.21.15.204:0/3466564482 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe950063860 con 0x7fe93c0a3cc0 2024-04-26T20:07:06.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe93c0a3cc0 msgr2=0x7fe93c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:06.451 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe93c0a3cc0 0x7fe93c0a6180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe94c0062a0 tx=0x7fe94c0061f0 comp rx=0 tx=0).stop 2024-04-26T20:07:06.459 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe95010b380 msgr2=0x7fe9501a2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:06.468 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe95010b380 0x7fe9501a2db0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe94800c930 tx=0x7fe94800ccf0 comp rx=0 tx=0).stop 2024-04-26T20:07:06.476 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 shutdown_connections 2024-04-26T20:07:06.486 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe95010b380 0x7fe9501a2db0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:06.497 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe93c0a3cc0 0x7fe93c0a6180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:06.506 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950075e80 0x7fe95019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:06.515 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 --2- 172.21.15.204:0/3466564482 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe950077450 0x7fe95019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:06.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 >> 172.21.15.204:0/3466564482 conn(0x7fe9500fea70 msgr2=0x7fe950107280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:06.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 shutdown_connections 2024-04-26T20:07:06.626 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:06.339+0000 7fe957d21700 1 -- 172.21.15.204:0/3466564482 wait complete. 2024-04-26T20:07:06.672 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:06.707 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:06.789 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:06.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:06.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:06.935 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:06 smithi204 ceph-mon[32618]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:06.942 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:06.949 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:06.956 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:06.963 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:06.971 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:06.979 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:06.986 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:06.992 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:07.000 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:06 smithi103 ceph-mon[32635]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:07.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:07.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:07.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:07.031 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:07.105 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:07.113 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:07.178 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:07.244 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:07.254 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:06 smithi094 ceph-mon[28050]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:07.262 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:07.271 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:07.279 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:07.288 DEBUG:teuthology.orchestra.run.smithi204:osd.11> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T20:07:07.299 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T20:07:07.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T20:07:07.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:07 smithi204 ceph-mon[32618]: from='osd.11 [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:07:07.657 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:07 smithi204 ceph-mon[32618]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:07:07.741 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:07 smithi094 ceph-mon[28050]: from='osd.11 [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:07:07.801 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:07 smithi094 ceph-mon[28050]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:07:07.816 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:07 smithi103 ceph-mon[32635]: from='osd.11 [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:07:07.907 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:07 smithi103 ceph-mon[32635]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:07:07.916 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:07:08.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:08 smithi103 ceph-mon[32635]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T20:07:08.826 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:08 smithi103 ceph-mon[32635]: from='osd.11 [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:07:08.894 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:08 smithi103 ceph-mon[32635]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T20:07:08.908 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:08.972 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:08 smithi103 ceph-mon[32635]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:07:08.986 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:08 smithi103 ceph-mon[32635]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:09.003 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:08 smithi204 ceph-mon[32618]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T20:07:09.015 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:08 smithi204 ceph-mon[32618]: from='osd.11 [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:07:09.028 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:08 smithi204 ceph-mon[32618]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T20:07:09.038 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:09.054 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:08 smithi204 ceph-mon[32618]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:07:09.068 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:08 smithi204 ceph-mon[32618]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:09.083 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:08 smithi094 ceph-mon[28050]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T20:07:09.097 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:08 smithi094 ceph-mon[28050]: from='osd.11 [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:07:09.111 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:08 smithi094 ceph-mon[28050]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T20:07:09.175 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:09.268 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:08 smithi094 ceph-mon[28050]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-26T20:07:09.280 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:08 smithi094 ceph-mon[28050]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:09.608 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:09 smithi204 ceph-mon[32618]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:07:09.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:09 smithi204 ceph-mon[32618]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T20:07:09.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:09.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:09 smithi204 ceph-mon[32618]: from='osd.11 ' entity='osd.11' 2024-04-26T20:07:09.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:09.610 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:07:09 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:07:09.286+0000 7feee4c77700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T20:07:09.610 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:07:09 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:07:09.302+0000 7feedca62700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:07:09.615 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:09 smithi094 ceph-mon[28050]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:07:09.616 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:09 smithi094 ceph-mon[28050]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T20:07:09.616 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:09.616 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:09 smithi094 ceph-mon[28050]: from='osd.11 ' entity='osd.11' 2024-04-26T20:07:09.616 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:09.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:09 smithi103 ceph-mon[32635]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-26T20:07:09.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:09 smithi103 ceph-mon[32635]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T20:07:09.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:09.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:09 smithi103 ceph-mon[32635]: from='osd.11 ' entity='osd.11' 2024-04-26T20:07:09.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:10.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.160+0000 7fb4fba00700 1 -- 172.21.15.94:0/4086959348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 msgr2=0x7fb4f410b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:10.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.160+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4086959348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 0x7fb4f410b8e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00b0d0 tx=0x7fb4ec00b3e0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.160+0000 7fb4fba00700 1 -- 172.21.15.94:0/4086959348 shutdown_connections 2024-04-26T20:07:10.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.160+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4086959348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 0x7fb4f410b8e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.160+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4086959348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f4103490 0x7fb4f41038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.160+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4086959348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4f4104690 0x7fb4f4104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.160+0000 7fb4fba00700 1 -- 172.21.15.94:0/4086959348 >> 172.21.15.94:0/4086959348 conn(0x7fb4f40fe9f0 msgr2=0x7fb4f4100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:10.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.160+0000 7fb4fba00700 1 -- 172.21.15.94:0/4086959348 shutdown_connections 2024-04-26T20:07:10.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.160+0000 7fb4fba00700 1 -- 172.21.15.94:0/4086959348 wait complete. 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.161+0000 7fb4fba00700 1 Processor -- start 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4fba00700 1 -- start start 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4fba00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f4103490 0x7fb4f419b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4fba00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4f4104690 0x7fb4f419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4fba00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 0x7fb4f41a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4fba00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f410ef10 con 0x7fb4f4104690 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4fba00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f410ed90 con 0x7fb4f4105030 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4fba00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f410f090 con 0x7fb4f4103490 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4f9a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 0x7fb4f41a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4f3e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4f4104690 0x7fb4f419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4f9a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 0x7fb4f41a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:36252/0 (socket says 172.21.15.94:36252) 2024-04-26T20:07:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4f9a00700 1 -- 172.21.15.94:0/4125577306 learned_addr learned my addr 172.21.15.94:0/4125577306 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:10.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.162+0000 7fb4f9000700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f4103490 0x7fb4f419b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:10.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.163+0000 7fb4f9000700 1 -- 172.21.15.94:0/4125577306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 msgr2=0x7fb4f41a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:10.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.163+0000 7fb4f9000700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 0x7fb4f41a2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.163+0000 7fb4f9000700 1 -- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4f4104690 msgr2=0x7fb4f419bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:10.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.163+0000 7fb4f9000700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4f4104690 0x7fb4f419bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.163+0000 7fb4f9000700 1 -- 172.21.15.94:0/4125577306 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f41a3380 con 0x7fb4f4103490 2024-04-26T20:07:10.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.163+0000 7fb4f3e00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4f4104690 0x7fb4f419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:07:10.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.163+0000 7fb4f9a00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 0x7fb4f41a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:07:10.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.163+0000 7fb4f9000700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f4103490 0x7fb4f419b770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800e9f0 tx=0x7fb4e800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:10.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.164+0000 7fb4f1600700 1 -- 172.21.15.94:0/4125577306 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e800cb70 con 0x7fb4f4103490 2024-04-26T20:07:10.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.164+0000 7fb4f1600700 1 -- 172.21.15.94:0/4125577306 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4e800ee90 con 0x7fb4f4103490 2024-04-26T20:07:10.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.164+0000 7fb4f1600700 1 -- 172.21.15.94:0/4125577306 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8018680 con 0x7fb4f4103490 2024-04-26T20:07:10.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.164+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f41a3670 con 0x7fb4f4103490 2024-04-26T20:07:10.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.164+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f41a3c00 con 0x7fb4f4103490 2024-04-26T20:07:10.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.166+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f406a110 con 0x7fb4f4103490 2024-04-26T20:07:10.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.167+0000 7fb4f1600700 1 -- 172.21.15.94:0/4125577306 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb4e801f020 con 0x7fb4f4103490 2024-04-26T20:07:10.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.168+0000 7fb4f1600700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4dc0a3cc0 0x7fb4dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:10.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.168+0000 7fb4f1600700 1 -- 172.21.15.94:0/4125577306 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7fb4e80cfbf0 con 0x7fb4f4103490 2024-04-26T20:07:10.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.168+0000 7fb4f3e00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4dc0a3cc0 0x7fb4dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:10.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.169+0000 7fb4f3e00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4dc0a3cc0 0x7fb4dc0a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009680 tx=0x7fb4e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:10.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.172+0000 7fb4f1600700 1 -- 172.21.15.94:0/4125577306 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb4e809caf0 con 0x7fb4f4103490 2024-04-26T20:07:10.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.427+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb4f41a4020 con 0x7fb4f4103490 2024-04-26T20:07:10.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.428+0000 7fb4f1600700 1 -- 172.21.15.94:0/4125577306 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7fb4e809c3f0 con 0x7fb4f4103490 2024-04-26T20:07:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:10.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.430+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4dc0a3cc0 msgr2=0x7fb4dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:10.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.430+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4dc0a3cc0 0x7fb4dc0a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009680 tx=0x7fb4e0009260 comp rx=0 tx=0).stop 2024-04-26T20:07:10.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.430+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f4103490 msgr2=0x7fb4f419b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:10.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.430+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f4103490 0x7fb4f419b770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800e9f0 tx=0x7fb4e800ed00 comp rx=0 tx=0).stop 2024-04-26T20:07:10.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.431+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 shutdown_connections 2024-04-26T20:07:10.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.431+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb4dc0a3cc0 0x7fb4dc0a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.431+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4f4105030 0x7fb4f41a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.431+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f4103490 0x7fb4f419b770 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.431+0000 7fb4fba00700 1 --2- 172.21.15.94:0/4125577306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4f4104690 0x7fb4f419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:10.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.431+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 >> 172.21.15.94:0/4125577306 conn(0x7fb4f40fe9f0 msgr2=0x7fb4f4075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:10.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.431+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 shutdown_connections 2024-04-26T20:07:10.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:10.431+0000 7fb4fba00700 1 -- 172.21.15.94:0/4125577306 wait complete. 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: purged_snaps scrub starts 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: purged_snaps scrub ok 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.728 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:10.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:10.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: osd.11 [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] boot 2024-04-26T20:07:10.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T20:07:10.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:10.729 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:10 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4125577306' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T20:07:10.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: purged_snaps scrub starts 2024-04-26T20:07:10.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: purged_snaps scrub ok 2024-04-26T20:07:10.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:10.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: osd.11 [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] boot 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:10.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:10 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4125577306' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T20:07:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: purged_snaps scrub starts 2024-04-26T20:07:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: purged_snaps scrub ok 2024-04-26T20:07:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:07:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:07:10.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:10.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:10.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:10.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: osd.11 [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] boot 2024-04-26T20:07:10.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T20:07:10.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:07:10.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:10 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4125577306' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T20:07:11.251 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1714162030,"num_in_osds":12,"osd_in_since":1714162009,"num_remapped_pgs":0} 2024-04-26T20:07:11.251 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T20:07:11.671 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:07:11.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:11 smithi103 ceph-mon[32635]: Detected new or changed devices on smithi204 2024-04-26T20:07:11.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:11 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi204 to 4272M 2024-04-26T20:07:11.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:11 smithi103 ceph-mon[32635]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T20:07:11.818 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:11 smithi094 ceph-mon[28050]: Detected new or changed devices on smithi204 2024-04-26T20:07:11.818 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:11 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi204 to 4272M 2024-04-26T20:07:11.818 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:11 smithi094 ceph-mon[28050]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T20:07:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:11 smithi204 ceph-mon[32618]: Detected new or changed devices on smithi204 2024-04-26T20:07:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:11 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi204 to 4272M 2024-04-26T20:07:11.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:11 smithi204 ceph-mon[32618]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T20:07:12.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:12 smithi103 ceph-mon[32635]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:12.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:12 smithi204 ceph-mon[32618]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:12.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:12 smithi094 ceph-mon[28050]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.873+0000 7f27f4a00700 1 -- 172.21.15.94:0/4036506021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f010b660 msgr2=0x7f27f010ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.873+0000 7f27f4a00700 1 --2- 172.21.15.94:0/4036506021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f010b660 0x7f27f010ba40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009990 tx=0x7f27e0009ca0 comp rx=0 tx=0).stop 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.873+0000 7f27f4a00700 1 -- 172.21.15.94:0/4036506021 shutdown_connections 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.873+0000 7f27f4a00700 1 --2- 172.21.15.94:0/4036506021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f0106020 0x7f27f0112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.873+0000 7f27f4a00700 1 --2- 172.21.15.94:0/4036506021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f010b660 0x7f27f010ba40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.873+0000 7f27f4a00700 1 --2- 172.21.15.94:0/4036506021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27f0105660 0x7f27f0105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.873+0000 7f27f4a00700 1 -- 172.21.15.94:0/4036506021 >> 172.21.15.94:0/4036506021 conn(0x7f27f0101150 msgr2=0x7f27f0103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.873+0000 7f27f4a00700 1 -- 172.21.15.94:0/4036506021 shutdown_connections 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.874+0000 7f27f4a00700 1 -- 172.21.15.94:0/4036506021 wait complete. 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.874+0000 7f27f4a00700 1 Processor -- start 2024-04-26T20:07:13.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27f4a00700 1 -- start start 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27f4a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27f0105660 0x7f27f01a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27f4a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f0106020 0x7f27f01a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27f4a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f010b660 0x7f27f019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27f4a00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0115890 con 0x7f27f0105660 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27f4a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0115710 con 0x7f27f0106020 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27f4a00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0115a10 con 0x7f27f010b660 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27e5400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f0106020 0x7f27f01a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27ede00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27f0105660 0x7f27f01a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27e5400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f0106020 0x7f27f01a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:57386/0 (socket says 172.21.15.94:57386) 2024-04-26T20:07:13.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.875+0000 7f27e5400700 1 -- 172.21.15.94:0/586436111 learned_addr learned my addr 172.21.15.94:0/586436111 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27ee800700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f010b660 0x7f27f019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27e5400700 1 -- 172.21.15.94:0/586436111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f010b660 msgr2=0x7f27f019bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27e5400700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f010b660 0x7f27f019bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27e5400700 1 -- 172.21.15.94:0/586436111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27f0105660 msgr2=0x7f27f01a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27e5400700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27f0105660 0x7f27f01a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27e5400700 1 -- 172.21.15.94:0/586436111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f019c450 con 0x7f27f0106020 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27ee800700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f010b660 0x7f27f019bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27e5400700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f0106020 0x7f27f01a1f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f27d800eb30 tx=0x7f27d800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27e7400700 1 -- 172.21.15.94:0/586436111 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d800cc90 con 0x7f27f0106020 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27e7400700 1 -- 172.21.15.94:0/586436111 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d80044d0 con 0x7f27f0106020 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f019c740 con 0x7f27f0106020 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27e7400700 1 -- 172.21.15.94:0/586436111 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8010430 con 0x7f27f0106020 2024-04-26T20:07:13.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.876+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f01a89d0 con 0x7f27f0106020 2024-04-26T20:07:13.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.877+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f010dbf0 con 0x7f27f0106020 2024-04-26T20:07:13.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.879+0000 7f27e7400700 1 -- 172.21.15.94:0/586436111 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f27d8003ea0 con 0x7f27f0106020 2024-04-26T20:07:13.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.881+0000 7f27e7400700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f27d00a3c40 0x7f27d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:13.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.881+0000 7f27e7400700 1 -- 172.21.15.94:0/586436111 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f27d80d0270 con 0x7f27f0106020 2024-04-26T20:07:13.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.881+0000 7f27ede00700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f27d00a3c40 0x7f27d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:13.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.881+0000 7f27ede00700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f27d00a3c40 0x7f27d00a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f27e0006010 tx=0x7f27e000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:13.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:13.885+0000 7f27e7400700 1 -- 172.21.15.94:0/586436111 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f27d809d050 con 0x7f27f0106020 2024-04-26T20:07:14.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.112+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27f019cfe0 con 0x7f27f0106020 2024-04-26T20:07:14.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.113+0000 7f27e7400700 1 -- 172.21.15.94:0/586436111 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18493 (secure 0 0 0) 0x7f27d809c950 con 0x7f27f0106020 2024-04-26T20:07:14.115 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:14.115 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":81,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:07:11.294222+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:07:14.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.116+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f27d00a3c40 msgr2=0x7f27d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:14.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f27d00a3c40 0x7f27d00a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f27e0006010 tx=0x7f27e000b260 comp rx=0 tx=0).stop 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f0106020 msgr2=0x7f27f01a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f0106020 0x7f27f01a1f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f27d800eb30 tx=0x7f27d800ee40 comp rx=0 tx=0).stop 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 shutdown_connections 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f010b660 0x7f27f019bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27f0105660 0x7f27f01a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f27d00a3c40 0x7f27d00a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 --2- 172.21.15.94:0/586436111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f0106020 0x7f27f01a1f90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 >> 172.21.15.94:0/586436111 conn(0x7f27f0101150 msgr2=0x7f27f01018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 shutdown_connections 2024-04-26T20:07:14.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:14.117+0000 7f27f4a00700 1 -- 172.21.15.94:0/586436111 wait complete. 2024-04-26T20:07:14.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:14 smithi204 ceph-mon[32618]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:14.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:14 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/586436111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:07:14.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:14 smithi094 ceph-mon[28050]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:14.894 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:14 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/586436111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:07:14.953 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T20:07:14.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T20:07:15.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:14 smithi103 ceph-mon[32635]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:15.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:14 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/586436111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:07:15.376 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:07:16.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:16 smithi094 ceph-mon[28050]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:16.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:16 smithi204 ceph-mon[32618]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:17.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:16 smithi103 ceph-mon[32635]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:17.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.676+0000 7f75a4000700 1 -- 172.21.15.94:0/1863715476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c1145f0 msgr2=0x7f759c1169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:17.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.676+0000 7f75a4000700 1 --2- 172.21.15.94:0/1863715476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c1145f0 0x7f759c1169e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f759400b0a0 tx=0x7f759400b3b0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.677+0000 7f75a4000700 1 -- 172.21.15.94:0/1863715476 shutdown_connections 2024-04-26T20:07:17.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.677+0000 7f75a4000700 1 --2- 172.21.15.94:0/1863715476 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c1145f0 0x7f759c1169e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.677+0000 7f75a4000700 1 --2- 172.21.15.94:0/1863715476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f759c075650 0x7f759c075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.677+0000 7f75a4000700 1 --2- 172.21.15.94:0/1863715476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f759c075f70 0x7f759c1140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.677+0000 7f75a4000700 1 -- 172.21.15.94:0/1863715476 >> 172.21.15.94:0/1863715476 conn(0x7f759c07a580 msgr2=0x7f759c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:17.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.677+0000 7f75a4000700 1 -- 172.21.15.94:0/1863715476 shutdown_connections 2024-04-26T20:07:17.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.677+0000 7f75a4000700 1 -- 172.21.15.94:0/1863715476 wait complete. 2024-04-26T20:07:17.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.678+0000 7f75a4000700 1 Processor -- start 2024-04-26T20:07:17.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.678+0000 7f75a4000700 1 -- start start 2024-04-26T20:07:17.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.678+0000 7f75a4000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f759c075650 0x7f759c1a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:17.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.678+0000 7f75a4000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c075f70 0x7f759c1a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.678+0000 7f75a4000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f759c1145f0 0x7f759c1a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.678+0000 7f75a4000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f759c119c20 con 0x7f759c075f70 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.678+0000 7f75a4000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f759c119aa0 con 0x7f759c1145f0 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.678+0000 7f75a4000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f759c119da0 con 0x7f759c075650 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a0c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c075f70 0x7f759c1a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a0c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c075f70 0x7f759c1a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54882/0 (socket says 172.21.15.94:54882) 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a0c00700 1 -- 172.21.15.94:0/3599012581 learned_addr learned my addr 172.21.15.94:0/3599012581 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a1600700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f759c075650 0x7f759c1a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a2000700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f759c1145f0 0x7f759c1a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a0c00700 1 -- 172.21.15.94:0/3599012581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f759c075650 msgr2=0x7f759c1a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a0c00700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f759c075650 0x7f759c1a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a0c00700 1 -- 172.21.15.94:0/3599012581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f759c1145f0 msgr2=0x7f759c1a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a0c00700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f759c1145f0 0x7f759c1a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a0c00700 1 -- 172.21.15.94:0/3599012581 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759c1a51c0 con 0x7f759c075f70 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a1600700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f759c075650 0x7f759c1a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.679+0000 7f75a0c00700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c075f70 0x7f759c1a06f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f758800eb30 tx=0x7f758800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.680+0000 7f759a000700 1 -- 172.21.15.94:0/3599012581 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758800cbd0 con 0x7f759c075f70 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.680+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f759c1a54b0 con 0x7f759c075f70 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.680+0000 7f759a000700 1 -- 172.21.15.94:0/3599012581 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75880044d0 con 0x7f759c075f70 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.680+0000 7f759a000700 1 -- 172.21.15.94:0/3599012581 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588003e30 con 0x7f759c075f70 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.680+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f759c1acd30 con 0x7f759c075f70 2024-04-26T20:07:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.681+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f759c06a110 con 0x7f759c075f70 2024-04-26T20:07:17.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.683+0000 7f759a000700 1 -- 172.21.15.94:0/3599012581 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7588004000 con 0x7f759c075f70 2024-04-26T20:07:17.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.684+0000 7f759a000700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75840a3c40 0x7f75840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:17.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.685+0000 7f759a000700 1 -- 172.21.15.94:0/3599012581 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f75880d02f0 con 0x7f759c075f70 2024-04-26T20:07:17.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.685+0000 7f75a1600700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75840a3c40 0x7f75840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:17.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.686+0000 7f75a1600700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75840a3c40 0x7f75840a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7590011490 tx=0x7f7590009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.689+0000 7f759a000700 1 -- 172.21.15.94:0/3599012581 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f75880042f0 con 0x7f759c075f70 2024-04-26T20:07:17.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.915+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f759c1acf60 con 0x7f759c075f70 2024-04-26T20:07:17.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.916+0000 7f759a000700 1 -- 172.21.15.94:0/3599012581 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f758809d0d0 con 0x7f759c075f70 2024-04-26T20:07:17.917 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:07:17.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.919+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75840a3c40 msgr2=0x7f75840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:17.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.919+0000 7f75a4000700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75840a3c40 0x7f75840a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7590011490 tx=0x7f7590009260 comp rx=0 tx=0).stop 2024-04-26T20:07:17.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.919+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c075f70 msgr2=0x7f759c1a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:17.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.919+0000 7f75a4000700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c075f70 0x7f759c1a06f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f758800eb30 tx=0x7f758800ee40 comp rx=0 tx=0).stop 2024-04-26T20:07:17.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.920+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 shutdown_connections 2024-04-26T20:07:17.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.920+0000 7f75a4000700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75840a3c40 0x7f75840a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.920+0000 7f75a4000700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f759c1145f0 0x7f759c1a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.920+0000 7f75a4000700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f759c075650 0x7f759c1a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.920+0000 7f75a4000700 1 --2- 172.21.15.94:0/3599012581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f759c075f70 0x7f759c1a06f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:17.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.920+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 >> 172.21.15.94:0/3599012581 conn(0x7f759c07a580 msgr2=0x7f759c110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:17.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.920+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 shutdown_connections 2024-04-26T20:07:17.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:17.920+0000 7f75a4000700 1 -- 172.21.15.94:0/3599012581 wait complete. 2024-04-26T20:07:18.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:18 smithi094 ceph-mon[28050]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:18.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:18 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3599012581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:07:18.819 INFO:tasks.cephadm:Adding mds.a on smithi094 2024-04-26T20:07:18.819 INFO:tasks.cephadm:Adding mds.d on smithi094 2024-04-26T20:07:18.819 INFO:tasks.cephadm:Adding mds.g on smithi094 2024-04-26T20:07:18.819 INFO:tasks.cephadm:Adding mds.j on smithi094 2024-04-26T20:07:18.819 INFO:tasks.cephadm:Adding mds.b on smithi103 2024-04-26T20:07:18.819 INFO:tasks.cephadm:Adding mds.e on smithi103 2024-04-26T20:07:18.820 INFO:tasks.cephadm:Adding mds.h on smithi103 2024-04-26T20:07:18.820 INFO:tasks.cephadm:Adding mds.k on smithi103 2024-04-26T20:07:18.820 INFO:tasks.cephadm:Adding mds.c on smithi204 2024-04-26T20:07:18.820 INFO:tasks.cephadm:Adding mds.f on smithi204 2024-04-26T20:07:18.820 INFO:tasks.cephadm:Adding mds.i on smithi204 2024-04-26T20:07:18.820 INFO:tasks.cephadm:Adding mds.l on smithi204 2024-04-26T20:07:18.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l' 2024-04-26T20:07:18.850 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:18 smithi204 ceph-mon[32618]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:18.850 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:18 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3599012581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:07:19.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:18 smithi103 ceph-mon[32635]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:19.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:18 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3599012581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:07:19.072 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:07:20.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.794+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/3983712381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74105020 msgr2=0x7f2c7410b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:20.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.794+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/3983712381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74105020 0x7f2c7410b8d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c009990 tx=0x7f2c6c009ca0 comp rx=0 tx=0).stop 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.794+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/3983712381 shutdown_connections 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.794+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/3983712381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74105020 0x7f2c7410b8d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.794+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/3983712381 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c74103480 0x7f2c741038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.794+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/3983712381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c74104680 0x7f2c74104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.794+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/3983712381 >> 172.21.15.204:0/3983712381 conn(0x7f2c740fea20 msgr2=0x7f2c74100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.794+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/3983712381 shutdown_connections 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.794+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/3983712381 wait complete. 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c7a0ad700 1 Processor -- start 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c7a0ad700 1 -- start start 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c7a0ad700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74103480 0x7f2c741995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c7a0ad700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c74104680 0x7f2c74199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c7a0ad700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c74105020 0x7f2c741a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:20.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c7a0ad700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7410efc0 con 0x7f2c74104680 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c7a0ad700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7410ee40 con 0x7f2c74105020 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c7a0ad700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7410f140 con 0x7f2c74103480 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c737fe700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74103480 0x7f2c741995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c737fe700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74103480 0x7f2c741995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:51700/0 (socket says 172.21.15.204:51700) 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c737fe700 1 -- 172.21.15.204:0/4055760623 learned_addr learned my addr 172.21.15.204:0/4055760623 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c73fff700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c74105020 0x7f2c741a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.795+0000 7f2c737fe700 1 -- 172.21.15.204:0/4055760623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c74105020 msgr2=0x7f2c741a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c737fe700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c74105020 0x7f2c741a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c737fe700 1 -- 172.21.15.204:0/4055760623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c74104680 msgr2=0x7f2c74199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c737fe700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c74104680 0x7f2c74199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c737fe700 1 -- 172.21.15.204:0/4055760623 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c741a1230 con 0x7f2c74103480 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c73fff700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c74105020 0x7f2c741a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c737fe700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74103480 0x7f2c741995d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400b9f0 tx=0x7f2c6400bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c70ff9700 1 -- 172.21.15.204:0/4055760623 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64019070 con 0x7f2c74103480 2024-04-26T20:07:20.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c70ff9700 1 -- 172.21.15.204:0/4055760623 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c64014440 con 0x7f2c74103480 2024-04-26T20:07:20.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c741a1520 con 0x7f2c74103480 2024-04-26T20:07:20.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c70ff9700 1 -- 172.21.15.204:0/4055760623 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64012580 con 0x7f2c74103480 2024-04-26T20:07:20.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.796+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c741a1a60 con 0x7f2c74103480 2024-04-26T20:07:20.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.797+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c7406a110 con 0x7f2c74103480 2024-04-26T20:07:20.801 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.798+0000 7f2c70ff9700 1 -- 172.21.15.204:0/4055760623 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2c6401c070 con 0x7f2c74103480 2024-04-26T20:07:20.801 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.799+0000 7f2c70ff9700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c5c0a3c40 0x7f2c5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:20.801 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.799+0000 7f2c70ff9700 1 -- 172.21.15.204:0/4055760623 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2c640cf360 con 0x7f2c74103480 2024-04-26T20:07:20.801 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.799+0000 7f2c72ffd700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c5c0a3c40 0x7f2c5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:20.801 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.800+0000 7f2c72ffd700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c5c0a3c40 0x7f2c5c0a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005950 tx=0x7f2c680058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:20.804 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:20.803+0000 7f2c70ff9700 1 -- 172.21.15.204:0/4055760623 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2c6409c1e0 con 0x7f2c74103480 2024-04-26T20:07:20.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:20 smithi204 ceph-mon[32618]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:20.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:20 smithi094 ceph-mon[28050]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:21.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.014+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}) -- 0x7f2c74063a60 con 0x7f2c5c0a3c40 2024-04-26T20:07:21.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.031+0000 7f2c70ff9700 1 -- 172.21.15.204:0/4055760623 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f2c74063a60 con 0x7f2c5c0a3c40 2024-04-26T20:07:21.032 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mds.all update... 2024-04-26T20:07:21.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:20 smithi103 ceph-mon[32635]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.033+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c5c0a3c40 msgr2=0x7f2c5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c5c0a3c40 0x7f2c5c0a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005950 tx=0x7f2c680058e0 comp rx=0 tx=0).stop 2024-04-26T20:07:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74103480 msgr2=0x7f2c741995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74103480 0x7f2c741995d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400b9f0 tx=0x7f2c6400bdb0 comp rx=0 tx=0).stop 2024-04-26T20:07:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 shutdown_connections 2024-04-26T20:07:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2c5c0a3c40 0x7f2c5c0a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c74105020 0x7f2c741a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c74103480 0x7f2c741995d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:21.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 --2- 172.21.15.204:0/4055760623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c74104680 0x7f2c74199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:21.036 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 >> 172.21.15.204:0/4055760623 conn(0x7f2c740fea20 msgr2=0x7f2c74107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:21.036 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 shutdown_connections 2024-04-26T20:07:21.036 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-26T20:07:21.034+0000 7f2c7a0ad700 1 -- 172.21.15.204:0/4055760623 wait complete. 2024-04-26T20:07:21.729 DEBUG:teuthology.orchestra.run.smithi094:mds.a> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T20:07:21.734 DEBUG:teuthology.orchestra.run.smithi094:mds.d> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T20:07:21.778 DEBUG:teuthology.orchestra.run.smithi094:mds.g> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T20:07:21.824 DEBUG:teuthology.orchestra.run.smithi094:mds.j> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T20:07:21.870 DEBUG:teuthology.orchestra.run.smithi103:mds.b> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T20:07:21.873 DEBUG:teuthology.orchestra.run.smithi103:mds.e> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T20:07:21.916 DEBUG:teuthology.orchestra.run.smithi103:mds.h> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T20:07:21.958 DEBUG:teuthology.orchestra.run.smithi103:mds.k> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T20:07:22.001 DEBUG:teuthology.orchestra.run.smithi204:mds.c> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T20:07:22.004 DEBUG:teuthology.orchestra.run.smithi204:mds.f> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T20:07:22.047 DEBUG:teuthology.orchestra.run.smithi204:mds.i> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T20:07:22.091 DEBUG:teuthology.orchestra.run.smithi204:mds.l> sudo journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T20:07:22.133 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T20:07:22.134 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:07:22.199 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:22 smithi094 ceph-mon[28050]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:07:22.200 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:22 smithi094 ceph-mon[28050]: Saving service mds.all spec with placement smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-04-26T20:07:22.200 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:22.200 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:22.200 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:22 smithi094 ceph-mon[28050]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:22.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:22 smithi103 ceph-mon[32635]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:07:22.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:22 smithi103 ceph-mon[32635]: Saving service mds.all spec with placement smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-04-26T20:07:22.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:22.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:22.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:22 smithi103 ceph-mon[32635]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:22.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:22 smithi204 ceph-mon[32618]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:07:22.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:22 smithi204 ceph-mon[32618]: Saving service mds.all spec with placement smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-04-26T20:07:22.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:22.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:07:22.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:22 smithi204 ceph-mon[32618]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:22.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.348+0000 7f55ae800640 1 -- 172.21.15.94:0/3331539145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 msgr2=0x7f55a81057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:22.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.348+0000 7f55ae800640 1 --2- 172.21.15.94:0/3331539145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 0x7f55a81057d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5598009900 tx=0x7f559802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.348+0000 7f55ae800640 1 -- 172.21.15.94:0/3331539145 shutdown_connections 2024-04-26T20:07:22.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.348+0000 7f55ae800640 1 --2- 172.21.15.94:0/3331539145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8108640 0x7f55a810aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.348+0000 7f55ae800640 1 --2- 172.21.15.94:0/3331539145 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55a8105d10 0x7f55a8108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.348+0000 7f55ae800640 1 --2- 172.21.15.94:0/3331539145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 0x7f55a81057d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.348+0000 7f55ae800640 1 -- 172.21.15.94:0/3331539145 >> 172.21.15.94:0/3331539145 conn(0x7f55a80fcdf0 msgr2=0x7f55a80ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:22.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.349+0000 7f55ae800640 1 -- 172.21.15.94:0/3331539145 shutdown_connections 2024-04-26T20:07:22.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.349+0000 7f55ae800640 1 -- 172.21.15.94:0/3331539145 wait complete. 2024-04-26T20:07:22.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.349+0000 7f55ae800640 1 Processor -- start 2024-04-26T20:07:22.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55ae800640 1 -- start start 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55ae800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 0x7f55a819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55ae800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8105d10 0x7f55a819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55ae800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55a8108640 0x7f55a81a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55ae800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810e500 con 0x7f55a8108640 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55ae800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810e380 con 0x7f55a8105d10 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55ae800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810e680 con 0x7f55a81033e0 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55a7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 0x7f55a819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55a7e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55a8108640 0x7f55a81a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55a7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 0x7f55a819add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:53358/0 (socket says 172.21.15.94:53358) 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f55a7400640 1 -- 172.21.15.94:0/2774299297 learned_addr learned my addr 172.21.15.94:0/2774299297 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.350+0000 7f559fe00640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8105d10 0x7f55a819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55a7400640 1 -- 172.21.15.94:0/2774299297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8105d10 msgr2=0x7f55a819b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55a7400640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8105d10 0x7f55a819b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55a7400640 1 -- 172.21.15.94:0/2774299297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55a8108640 msgr2=0x7f55a81a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55a7400640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55a8108640 0x7f55a81a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55a7400640 1 -- 172.21.15.94:0/2774299297 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a81a29f0 con 0x7f55a81033e0 2024-04-26T20:07:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55a7400640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 0x7f55a819add0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5598005ab0 tx=0x7f5598003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55a4c00640 1 -- 172.21.15.94:0/2774299297 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559803f070 con 0x7f55a81033e0 2024-04-26T20:07:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55a4c00640 1 -- 172.21.15.94:0/2774299297 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5598031c50 con 0x7f55a81033e0 2024-04-26T20:07:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a81a2c80 con 0x7f55a81033e0 2024-04-26T20:07:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55a4c00640 1 -- 172.21.15.94:0/2774299297 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5598035030 con 0x7f55a81033e0 2024-04-26T20:07:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.351+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a81a3170 con 0x7f55a81033e0 2024-04-26T20:07:22.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.352+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a810a190 con 0x7f55a81033e0 2024-04-26T20:07:22.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.354+0000 7f55a4c00640 1 -- 172.21.15.94:0/2774299297 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5598042070 con 0x7f55a81033e0 2024-04-26T20:07:22.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.356+0000 7f55a4c00640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f558c0a3bb0 0x7f558c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:22.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.356+0000 7f55a4c00640 1 -- 172.21.15.94:0/2774299297 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f55980f0e60 con 0x7f55a81033e0 2024-04-26T20:07:22.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.356+0000 7f559fe00640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f558c0a3bb0 0x7f558c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:22.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.357+0000 7f559fe00640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f558c0a3bb0 0x7f558c0a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55900095f0 tx=0x7f5590005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:22.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.358+0000 7f55a4c00640 1 -- 172.21.15.94:0/2774299297 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f55980bd5f0 con 0x7f55a81033e0 2024-04-26T20:07:22.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.560+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55a802dca0 con 0x7f55a81033e0 2024-04-26T20:07:22.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.561+0000 7f55a4c00640 1 -- 172.21.15.94:0/2774299297 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18493 (secure 0 0 0) 0x7f55980c18f0 con 0x7f55a81033e0 2024-04-26T20:07:22.563 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:22.563 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":81,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:07:11.294222+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:07:22.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f558c0a3bb0 msgr2=0x7f558c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:22.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f558c0a3bb0 0x7f558c0a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55900095f0 tx=0x7f5590005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:22.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 msgr2=0x7f55a819add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 0x7f55a819add0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5598005ab0 tx=0x7f5598003b90 comp rx=0 tx=0).stop 2024-04-26T20:07:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 shutdown_connections 2024-04-26T20:07:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f558c0a3bb0 0x7f558c0a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55a8108640 0x7f55a81a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8105d10 0x7f55a819b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 --2- 172.21.15.94:0/2774299297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55a81033e0 0x7f55a819add0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 >> 172.21.15.94:0/2774299297 conn(0x7f55a80fcdf0 msgr2=0x7f55a8102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 shutdown_connections 2024-04-26T20:07:22.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.564+0000 7f55ae800640 1 -- 172.21.15.94:0/2774299297 wait complete. 2024-04-26T20:07:22.587 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T20:07:22.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:07:22.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.793+0000 7f6467e00640 1 -- 172.21.15.94:0/1420397101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078530 msgr2=0x7f6468078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:22.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.793+0000 7f6467e00640 1 --2- 172.21.15.94:0/1420397101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078530 0x7f6468078990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6450009900 tx=0x7f645002a410 comp rx=0 tx=0).stop 2024-04-26T20:07:22.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.794+0000 7f6467e00640 1 -- 172.21.15.94:0/1420397101 shutdown_connections 2024-04-26T20:07:22.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.794+0000 7f6467e00640 1 --2- 172.21.15.94:0/1420397101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6468078ed0 0x7f6468079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.794+0000 7f6467e00640 1 --2- 172.21.15.94:0/1420397101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078530 0x7f6468078990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.794+0000 7f6467e00640 1 --2- 172.21.15.94:0/1420397101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64680772f0 0x7f6468077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.794+0000 7f6467e00640 1 -- 172.21.15.94:0/1420397101 >> 172.21.15.94:0/1420397101 conn(0x7f64680fffe0 msgr2=0x7f6468102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:22.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.794+0000 7f6467e00640 1 -- 172.21.15.94:0/1420397101 shutdown_connections 2024-04-26T20:07:22.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.794+0000 7f6467e00640 1 -- 172.21.15.94:0/1420397101 wait complete. 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467e00640 1 Processor -- start 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467e00640 1 -- start start 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64680772f0 0x7f646819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6468078530 0x7f646819f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078ed0 0x7f64681a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467e00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f64681149b0 con 0x7f6468078ed0 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6468114830 con 0x7f6468078530 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6468114b30 con 0x7f64680772f0 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078ed0 0x7f64681a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:22.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6466a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64680772f0 0x7f646819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078ed0 0x7f64681a66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56426/0 (socket says 172.21.15.94:56426) 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f6467400640 1 -- 172.21.15.94:0/4144046855 learned_addr learned my addr 172.21.15.94:0/4144046855 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.795+0000 7f645e000640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6468078530 0x7f646819f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f6467400640 1 -- 172.21.15.94:0/4144046855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64680772f0 msgr2=0x7f646819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f6467400640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64680772f0 0x7f646819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f6467400640 1 -- 172.21.15.94:0/4144046855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6468078530 msgr2=0x7f646819f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f6467400640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6468078530 0x7f646819f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f6467400640 1 -- 172.21.15.94:0/4144046855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64681a6d10 con 0x7f6468078ed0 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f6467400640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078ed0 0x7f64681a66a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f646000eb10 tx=0x7f646000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f645fe00640 1 -- 172.21.15.94:0/4144046855 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460004b20 con 0x7f6468078ed0 2024-04-26T20:07:22.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f645fe00640 1 -- 172.21.15.94:0/4144046855 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6460004cc0 con 0x7f6468078ed0 2024-04-26T20:07:22.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64681a6fa0 con 0x7f6468078ed0 2024-04-26T20:07:22.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f645fe00640 1 -- 172.21.15.94:0/4144046855 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460010670 con 0x7f6468078ed0 2024-04-26T20:07:22.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.796+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64681a7470 con 0x7f6468078ed0 2024-04-26T20:07:22.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.797+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6468111670 con 0x7f6468078ed0 2024-04-26T20:07:22.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.800+0000 7f645fe00640 1 -- 172.21.15.94:0/4144046855 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6460005020 con 0x7f6468078ed0 2024-04-26T20:07:22.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.801+0000 7f645fe00640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f644c0a3bb0 0x7f644c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:22.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.801+0000 7f645fe00640 1 -- 172.21.15.94:0/4144046855 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f64600d0560 con 0x7f6468078ed0 2024-04-26T20:07:22.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.801+0000 7f6466a00640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f644c0a3bb0 0x7f644c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:22.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.802+0000 7f6466a00640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f644c0a3bb0 0x7f644c0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6458009bd0 tx=0x7f6458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:22.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:22.806+0000 7f645fe00640 1 -- 172.21.15.94:0/4144046855 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f646009d340 con 0x7f6468078ed0 2024-04-26T20:07:23.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.007+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6468063530 con 0x7f6468078ed0 2024-04-26T20:07:23.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.008+0000 7f645fe00640 1 -- 172.21.15.94:0/4144046855 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f646009cc40 con 0x7f6468078ed0 2024-04-26T20:07:23.009 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:07:23.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f644c0a3bb0 msgr2=0x7f644c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:23.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f644c0a3bb0 0x7f644c0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6458009bd0 tx=0x7f6458005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078ed0 msgr2=0x7f64681a66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078ed0 0x7f64681a66a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f646000eb10 tx=0x7f646000c2d0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 shutdown_connections 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f644c0a3bb0 0x7f644c0a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6468078ed0 0x7f64681a66a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6468078530 0x7f646819f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 --2- 172.21.15.94:0/4144046855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64680772f0 0x7f646819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 >> 172.21.15.94:0/4144046855 conn(0x7f64680fffe0 msgr2=0x7f64681017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 shutdown_connections 2024-04-26T20:07:23.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.011+0000 7f6467e00640 1 -- 172.21.15.94:0/4144046855 wait complete. 2024-04-26T20:07:23.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:23.222 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:23.222 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:23.222 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:23.222 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:23.222 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:23.222 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:23.222 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:23 smithi204 ceph-mon[32618]: Deploying daemon mds.l on smithi204 2024-04-26T20:07:23.223 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2774299297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:07:23.223 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4144046855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:07:23.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.230+0000 7efce5600640 1 -- 172.21.15.94:0/2957858984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0105b80 msgr2=0x7efce0106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:23.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.230+0000 7efce5600640 1 --2- 172.21.15.94:0/2957858984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0105b80 0x7efce0106000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009900 tx=0x7efcc802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.231+0000 7efce5600640 1 -- 172.21.15.94:0/2957858984 shutdown_connections 2024-04-26T20:07:23.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.231+0000 7efce5600640 1 --2- 172.21.15.94:0/2957858984 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efce0106540 0x7efce010cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.231+0000 7efce5600640 1 --2- 172.21.15.94:0/2957858984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0105b80 0x7efce0106000 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.231+0000 7efce5600640 1 --2- 172.21.15.94:0/2957858984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efce0104990 0x7efce0104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.231+0000 7efce5600640 1 -- 172.21.15.94:0/2957858984 >> 172.21.15.94:0/2957858984 conn(0x7efce00fff00 msgr2=0x7efce0102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:23.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.231+0000 7efce5600640 1 -- 172.21.15.94:0/2957858984 shutdown_connections 2024-04-26T20:07:23.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.231+0000 7efce5600640 1 -- 172.21.15.94:0/2957858984 wait complete. 2024-04-26T20:07:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.232+0000 7efce5600640 1 Processor -- start 2024-04-26T20:07:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.232+0000 7efce5600640 1 -- start start 2024-04-26T20:07:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.232+0000 7efce5600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0104990 0x7efce019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.232+0000 7efce5600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efce0105b80 0x7efce019b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.232+0000 7efce5600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efce0106540 0x7efce01a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.232+0000 7efce5600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7efce01104a0 con 0x7efce0106540 2024-04-26T20:07:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.232+0000 7efce5600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efce0110320 con 0x7efce0105b80 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.232+0000 7efce5600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efce0110620 con 0x7efce0104990 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.232+0000 7efcdfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0104990 0x7efce019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0104990 0x7efce019ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:53404/0 (socket says 172.21.15.94:53404) 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdfe00640 1 -- 172.21.15.94:0/3318999907 learned_addr learned my addr 172.21.15.94:0/3318999907 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efce4c00640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efce0106540 0x7efce01a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdf400640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efce0105b80 0x7efce019b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdfe00640 1 -- 172.21.15.94:0/3318999907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efce0105b80 msgr2=0x7efce019b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdfe00640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efce0105b80 0x7efce019b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdfe00640 1 -- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efce0106540 msgr2=0x7efce01a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdfe00640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efce0106540 0x7efce01a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdfe00640 1 -- 172.21.15.94:0/3318999907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce01a2760 con 0x7efce0104990 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efce4c00640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efce0106540 0x7efce01a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdf400640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efce0105b80 0x7efce019b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:07:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdfe00640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0104990 0x7efce019ab40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efcd000ebc0 tx=0x7efcd000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdcc00640 1 -- 172.21.15.94:0/3318999907 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0003b70 con 0x7efce0104990 2024-04-26T20:07:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdcc00640 1 -- 172.21.15.94:0/3318999907 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcd0003d10 con 0x7efce0104990 2024-04-26T20:07:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efcdcc00640 1 -- 172.21.15.94:0/3318999907 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0019800 con 0x7efce0104990 2024-04-26T20:07:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.233+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce01a2a50 con 0x7efce0104990 2024-04-26T20:07:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.234+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce01a2fa0 con 0x7efce0104990 2024-04-26T20:07:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.235+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce010bd30 con 0x7efce0104990 2024-04-26T20:07:23.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.237+0000 7efcdcc00640 1 -- 172.21.15.94:0/3318999907 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efcd0004b60 con 0x7efce0104990 2024-04-26T20:07:23.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.238+0000 7efcdcc00640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcc40a3bb0 0x7efcc40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:23.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.238+0000 7efcdcc00640 1 -- 172.21.15.94:0/3318999907 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efcd00d0ba0 con 0x7efce0104990 2024-04-26T20:07:23.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.239+0000 7efcdf400640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcc40a3bb0 0x7efcc40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:23.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.239+0000 7efcdf400640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcc40a3bb0 0x7efcc40a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efce019c050 tx=0x7efcc80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:23.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.240+0000 7efcdcc00640 1 -- 172.21.15.94:0/3318999907 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efcd00d0e20 con 0x7efce0104990 2024-04-26T20:07:23.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:23.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:23.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:23.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:23.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:23.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:23.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:23 smithi103 ceph-mon[32635]: Deploying daemon mds.l on smithi204 2024-04-26T20:07:23.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2774299297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:07:23.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4144046855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:07:23.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.483+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efce002dad0 con 0x7efce0104990 2024-04-26T20:07:23.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.484+0000 7efcdcc00640 1 -- 172.21.15.94:0/3318999907 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7efcd009d980 con 0x7efce0104990 2024-04-26T20:07:23.485 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:23.486 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-26T20:07:23.486 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 1 2024-04-26T20:07:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:07:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:23 smithi094 ceph-mon[28050]: Deploying daemon mds.l on smithi204 2024-04-26T20:07:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2774299297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:07:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4144046855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:07:23.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcc40a3bb0 msgr2=0x7efcc40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:23.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcc40a3bb0 0x7efcc40a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efce019c050 tx=0x7efcc80314a0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0104990 msgr2=0x7efce019ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:23.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0104990 0x7efce019ab40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efcd000ebc0 tx=0x7efcd000c6a0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 shutdown_connections 2024-04-26T20:07:23.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7efcc40a3bb0 0x7efcc40a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efce0106540 0x7efce01a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efce0105b80 0x7efce019b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 --2- 172.21.15.94:0/3318999907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efce0104990 0x7efce019ab40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:23.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 >> 172.21.15.94:0/3318999907 conn(0x7efce00fff00 msgr2=0x7efce0108f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:23.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 shutdown_connections 2024-04-26T20:07:23.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:23.487+0000 7efce5600640 1 -- 172.21.15.94:0/3318999907 wait complete. 2024-04-26T20:07:24.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:24 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3318999907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:24.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:24 smithi204 ceph-mon[32618]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:24 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3318999907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:24 smithi103 ceph-mon[32635]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:24.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:24 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3318999907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:24.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:24 smithi094 ceph-mon[28050]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:24.553 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:07:24 smithi204 systemd[1]: Starting Ceph mds.l for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:07:24.874 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:07:24 smithi204 podman[62557]: 2024-04-26 20:07:24.553121205 +0000 UTC m=+0.110368150 container create ccaadfeb5f0e07a407008e2941728791dd2f4bf3c09d1be02d3ad8e1c9e56437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:07:24.874 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:07:24 smithi204 podman[62557]: 2024-04-26 20:07:24.460826376 +0000 UTC m=+0.018073349 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:07:24.874 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:07:24 smithi204 podman[62557]: 2024-04-26 20:07:24.773119662 +0000 UTC m=+0.330366618 container init ccaadfeb5f0e07a407008e2941728791dd2f4bf3c09d1be02d3ad8e1c9e56437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:07:24.875 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:07:24 smithi204 podman[62557]: 2024-04-26 20:07:24.775735631 +0000 UTC m=+0.332982577 container start ccaadfeb5f0e07a407008e2941728791dd2f4bf3c09d1be02d3ad8e1c9e56437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:07:24.875 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:07:24 smithi204 ceph-mds[62576]: -- 172.21.15.204:0/2124094737 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba014f3a40 con 0x55ba014c7000 2024-04-26T20:07:24.875 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:07:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-l[62572]: starting mds.l at 2024-04-26T20:07:24.875 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:07:24 smithi204 bash[62557]: ccaadfeb5f0e07a407008e2941728791dd2f4bf3c09d1be02d3ad8e1c9e56437 2024-04-26T20:07:25.131 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:07:24 smithi204 systemd[1]: Started Ceph mds.l for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:25.508 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.713+0000 7fd9a3200640 1 -- 172.21.15.94:0/1677618157 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd99c108f90 msgr2=0x7fd99c1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.713+0000 7fd9a3200640 1 --2- 172.21.15.94:0/1677618157 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd99c108f90 0x7fd99c1093b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009900 tx=0x7fd98c02a410 comp rx=0 tx=0).stop 2024-04-26T20:07:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.713+0000 7fd9a3200640 1 -- 172.21.15.94:0/1677618157 shutdown_connections 2024-04-26T20:07:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.713+0000 7fd9a3200640 1 --2- 172.21.15.94:0/1677618157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c10ab40 0x7fd99c1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.713+0000 7fd9a3200640 1 --2- 172.21.15.94:0/1677618157 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd99c10a180 0x7fd99c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.713+0000 7fd9a3200640 1 --2- 172.21.15.94:0/1677618157 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd99c108f90 0x7fd99c1093b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.713+0000 7fd9a3200640 1 -- 172.21.15.94:0/1677618157 >> 172.21.15.94:0/1677618157 conn(0x7fd99c077f40 msgr2=0x7fd99c07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.714+0000 7fd9a3200640 1 -- 172.21.15.94:0/1677618157 shutdown_connections 2024-04-26T20:07:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.714+0000 7fd9a3200640 1 -- 172.21.15.94:0/1677618157 wait complete. 2024-04-26T20:07:25.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.714+0000 7fd9a3200640 1 Processor -- start 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd9a3200640 1 -- start start 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd9a3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd99c108f90 0x7fd99c19f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd9a3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c10a180 0x7fd99c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd9a3200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd99c10ab40 0x7fd99c1a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd9a3200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c114ad0 con 0x7fd99c10ab40 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd9a3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c114950 con 0x7fd99c10a180 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd9a3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c114c50 con 0x7fd99c108f90 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd9a1200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd99c10ab40 0x7fd99c1a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd993400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c10a180 0x7fd99c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd993400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c10a180 0x7fd99c19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:33416/0 (socket says 172.21.15.94:33416) 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd993400640 1 -- 172.21.15.94:0/2710545762 learned_addr learned my addr 172.21.15.94:0/2710545762 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.715+0000 7fd99be00640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd99c108f90 0x7fd99c19f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:25.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd993400640 1 -- 172.21.15.94:0/2710545762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd99c108f90 msgr2=0x7fd99c19f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd993400640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd99c108f90 0x7fd99c19f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd993400640 1 -- 172.21.15.94:0/2710545762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd99c10ab40 msgr2=0x7fd99c1a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd993400640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd99c10ab40 0x7fd99c1a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd993400640 1 -- 172.21.15.94:0/2710545762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c1a6e00 con 0x7fd99c10a180 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd993400640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c10a180 0x7fd99c19f720 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd98400e9f0 tx=0x7fd98400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd999600640 1 -- 172.21.15.94:0/2710545762 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98400ccc0 con 0x7fd99c10a180 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd999600640 1 -- 172.21.15.94:0/2710545762 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd98400ce60 con 0x7fd99c10a180 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd999600640 1 -- 172.21.15.94:0/2710545762 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984010430 con 0x7fd99c10a180 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99c1a70f0 con 0x7fd99c10a180 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.716+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99c1a7640 con 0x7fd99c10a180 2024-04-26T20:07:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.719+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd99c110330 con 0x7fd99c10a180 2024-04-26T20:07:25.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.720+0000 7fd999600640 1 -- 172.21.15.94:0/2710545762 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd9840046c0 con 0x7fd99c10a180 2024-04-26T20:07:25.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.721+0000 7fd999600640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9800a3c30 0x7fd9800a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:25.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.721+0000 7fd999600640 1 -- 172.21.15.94:0/2710545762 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd9840d0430 con 0x7fd99c10a180 2024-04-26T20:07:25.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.721+0000 7fd99be00640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9800a3c30 0x7fd9800a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:25.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.722+0000 7fd99be00640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9800a3c30 0x7fd9800a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd98c000c00 tx=0x7fd98c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.726+0000 7fd999600640 1 -- 172.21.15.94:0/2710545762 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd98409d210 con 0x7fd99c10a180 2024-04-26T20:07:25.958 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.204:6832/3414557921,v1:172.21.15.204:6833/3414557921] up:boot 2024-04-26T20:07:25.958 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: fsmap 1 up:standby 2024-04-26T20:07:25.958 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T20:07:25.958 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:25.958 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:25.958 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:25.959 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:25.959 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:25.959 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:25.959 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: Deploying daemon mds.k on smithi103 2024-04-26T20:07:25.959 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:25 smithi103 ceph-mon[32635]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:25.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.976+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd99c02dad0 con 0x7fd99c10a180 2024-04-26T20:07:25.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.977+0000 7fd999600640 1 -- 172.21.15.94:0/2710545762 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fd98409cb10 con 0x7fd99c10a180 2024-04-26T20:07:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:25.979 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T20:07:25.979 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 2 2024-04-26T20:07:25.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9800a3c30 msgr2=0x7fd9800a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:25.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9800a3c30 0x7fd9800a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd98c000c00 tx=0x7fd98c0314a0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c10a180 msgr2=0x7fd99c19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c10a180 0x7fd99c19f720 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd98400e9f0 tx=0x7fd98400edf0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 shutdown_connections 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9800a3c30 0x7fd9800a60f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd99c10ab40 0x7fd99c1a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd99c10a180 0x7fd99c19f720 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 --2- 172.21.15.94:0/2710545762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd99c108f90 0x7fd99c19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 >> 172.21.15.94:0/2710545762 conn(0x7fd99c077f40 msgr2=0x7fd99c10d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 shutdown_connections 2024-04-26T20:07:25.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:25.980+0000 7fd9a3200640 1 -- 172.21.15.94:0/2710545762 wait complete. 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.204:6832/3414557921,v1:172.21.15.204:6833/3414557921] up:boot 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: fsmap 1 up:standby 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: Deploying daemon mds.k on smithi103 2024-04-26T20:07:25.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:25 smithi094 ceph-mon[28050]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.204:6832/3414557921,v1:172.21.15.204:6833/3414557921] up:boot 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: fsmap 1 up:standby 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: Deploying daemon mds.k on smithi103 2024-04-26T20:07:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:25 smithi204 ceph-mon[32618]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:26 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2710545762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:27.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:26 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2710545762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:27.285 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:07:27 smithi103 systemd[1]: Starting Ceph mds.k for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:07:27.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:26 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2710545762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:27.602 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:07:27 smithi103 podman[62269]: 2024-04-26 20:07:27.234254842 +0000 UTC m=+0.018557539 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:07:27.602 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:07:27 smithi103 podman[62269]: 2024-04-26 20:07:27.348777775 +0000 UTC m=+0.133080451 container create 93948c78258d3dd6863350cb784d49c4271f115946ee8958d8afdb322dd0a10a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:07:27.602 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:07:27 smithi103 podman[62269]: 2024-04-26 20:07:27.601443193 +0000 UTC m=+0.385745866 container init 93948c78258d3dd6863350cb784d49c4271f115946ee8958d8afdb322dd0a10a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:07:27.978 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:07:27 smithi103 podman[62269]: 2024-04-26 20:07:27.60477187 +0000 UTC m=+0.389074551 container start 93948c78258d3dd6863350cb784d49c4271f115946ee8958d8afdb322dd0a10a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-k, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:07:27.979 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:07:27 smithi103 ceph-mds[62288]: -- 172.21.15.103:0/1228985969 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5651a74fba40 con 0x5651a74cf000 2024-04-26T20:07:27.979 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:07:27 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-k[62284]: starting mds.k at 2024-04-26T20:07:27.979 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:07:27 smithi103 bash[62269]: 93948c78258d3dd6863350cb784d49c4271f115946ee8958d8afdb322dd0a10a 2024-04-26T20:07:27.979 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:07:27 smithi103 systemd[1]: Started Ceph mds.k for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:28.003 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.183+0000 7f1182800640 1 -- 172.21.15.94:0/4222225056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 msgr2=0x7f117c102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.183+0000 7f1182800640 1 --2- 172.21.15.94:0/4222225056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 0x7f117c102da0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f116c009860 tx=0x7f116c02a340 comp rx=0 tx=0).stop 2024-04-26T20:07:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.183+0000 7f1182800640 1 -- 172.21.15.94:0/4222225056 shutdown_connections 2024-04-26T20:07:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.183+0000 7f1182800640 1 --2- 172.21.15.94:0/4222225056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117c104530 0x7f117c10adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.183+0000 7f1182800640 1 --2- 172.21.15.94:0/4222225056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f117c103b70 0x7f117c103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.183+0000 7f1182800640 1 --2- 172.21.15.94:0/4222225056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 0x7f117c102da0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.183+0000 7f1182800640 1 -- 172.21.15.94:0/4222225056 >> 172.21.15.94:0/4222225056 conn(0x7f117c0fdf10 msgr2=0x7f117c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.183+0000 7f1182800640 1 -- 172.21.15.94:0/4222225056 shutdown_connections 2024-04-26T20:07:28.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.183+0000 7f1182800640 1 -- 172.21.15.94:0/4222225056 wait complete. 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.184+0000 7f1182800640 1 Processor -- start 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.184+0000 7f1182800640 1 -- start start 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f1182800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 0x7f117c19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f1182800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f117c103b70 0x7f117c19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f1182800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117c104530 0x7f117c1a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f1182800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f117c10e4a0 con 0x7f117c102980 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f1182800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f117c10e320 con 0x7f117c104530 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f1182800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f117c10e620 con 0x7f117c103b70 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 0x7f117c19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:28.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 0x7f117c19ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56470/0 (socket says 172.21.15.94:56470) 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117b400640 1 -- 172.21.15.94:0/1137275032 learned_addr learned my addr 172.21.15.94:0/1137275032 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117be00640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117c104530 0x7f117c1a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117aa00640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f117c103b70 0x7f117c19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117b400640 1 -- 172.21.15.94:0/1137275032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f117c103b70 msgr2=0x7f117c19b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117b400640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f117c103b70 0x7f117c19b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117b400640 1 -- 172.21.15.94:0/1137275032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117c104530 msgr2=0x7f117c1a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117b400640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117c104530 0x7f117c1a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.185+0000 7f117b400640 1 -- 172.21.15.94:0/1137275032 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117c1a2a80 con 0x7f117c102980 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.186+0000 7f117aa00640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f117c103b70 0x7f117c19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.186+0000 7f117be00640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117c104530 0x7f117c1a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.186+0000 7f117b400640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 0x7f117c19ae60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f116c0057c0 tx=0x7f116c032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.186+0000 7f1173e00640 1 -- 172.21.15.94:0/1137275032 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c037040 con 0x7f117c102980 2024-04-26T20:07:28.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.186+0000 7f1173e00640 1 -- 172.21.15.94:0/1137275032 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f116c030bd0 con 0x7f117c102980 2024-04-26T20:07:28.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.186+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f117c1a2d10 con 0x7f117c102980 2024-04-26T20:07:28.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.186+0000 7f1173e00640 1 -- 172.21.15.94:0/1137275032 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c005280 con 0x7f117c102980 2024-04-26T20:07:28.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.186+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f117c1a31a0 con 0x7f117c102980 2024-04-26T20:07:28.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.188+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f117c1a3430 con 0x7f117c102980 2024-04-26T20:07:28.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.189+0000 7f1173e00640 1 -- 172.21.15.94:0/1137275032 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f116c043020 con 0x7f117c102980 2024-04-26T20:07:28.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.191+0000 7f1173e00640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11600a3c30 0x7f11600a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:28.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.191+0000 7f1173e00640 1 -- 172.21.15.94:0/1137275032 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f116c0ef590 con 0x7f117c102980 2024-04-26T20:07:28.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.191+0000 7f117aa00640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11600a3c30 0x7f11600a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:28.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.192+0000 7f117aa00640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11600a3c30 0x7f11600a60f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f117c19c370 tx=0x7f1168009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:28.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.196+0000 7f1173e00640 1 -- 172.21.15.94:0/1137275032 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f116c030d80 con 0x7f117c102980 2024-04-26T20:07:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:27 smithi094 ceph-mon[28050]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:27 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:27 smithi103 ceph-mon[32635]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:27 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:28.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:27 smithi204 ceph-mon[32618]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:28.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:28.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:28.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:28.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:28.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:28.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:27 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:28.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.455+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f117c1a37c0 con 0x7f117c102980 2024-04-26T20:07:28.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.456+0000 7f1173e00640 1 -- 172.21.15.94:0/1137275032 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f116c0bc370 con 0x7f117c102980 2024-04-26T20:07:28.457 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:28.457 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T20:07:28.457 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 3 2024-04-26T20:07:28.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11600a3c30 msgr2=0x7f11600a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:28.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11600a3c30 0x7f11600a60f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f117c19c370 tx=0x7f1168009260 comp rx=0 tx=0).stop 2024-04-26T20:07:28.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 msgr2=0x7f117c19ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:28.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 0x7f117c19ae60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f116c0057c0 tx=0x7f116c032da0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 shutdown_connections 2024-04-26T20:07:28.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11600a3c30 0x7f11600a60f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f117c104530 0x7f117c1a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f117c103b70 0x7f117c19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 --2- 172.21.15.94:0/1137275032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f117c102980 0x7f117c19ae60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:28.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.458+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 >> 172.21.15.94:0/1137275032 conn(0x7f117c0fdf10 msgr2=0x7f117c1093b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:28.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.459+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 shutdown_connections 2024-04-26T20:07:28.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:28.459+0000 7f1182800640 1 -- 172.21.15.94:0/1137275032 wait complete. 2024-04-26T20:07:29.149 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:28 smithi094 ceph-mon[28050]: Deploying daemon mds.j on smithi094 2024-04-26T20:07:29.149 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:28 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.103:6832/695386981,v1:172.21.15.103:6833/695386981] up:boot 2024-04-26T20:07:29.149 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:28 smithi094 ceph-mon[28050]: fsmap 2 up:standby 2024-04-26T20:07:29.149 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:28 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T20:07:29.149 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:28 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1137275032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:29.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:28 smithi103 ceph-mon[32635]: Deploying daemon mds.j on smithi094 2024-04-26T20:07:29.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:28 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.103:6832/695386981,v1:172.21.15.103:6833/695386981] up:boot 2024-04-26T20:07:29.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:28 smithi103 ceph-mon[32635]: fsmap 2 up:standby 2024-04-26T20:07:29.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:28 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T20:07:29.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:28 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1137275032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:29.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:28 smithi204 ceph-mon[32618]: Deploying daemon mds.j on smithi094 2024-04-26T20:07:29.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:28 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.103:6832/695386981,v1:172.21.15.103:6833/695386981] up:boot 2024-04-26T20:07:29.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:28 smithi204 ceph-mon[32618]: fsmap 2 up:standby 2024-04-26T20:07:29.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:28 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T20:07:29.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:28 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1137275032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:30.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:29 smithi103 ceph-mon[32635]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:30.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:29 smithi204 ceph-mon[32618]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:30.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:30.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:29 smithi094 ceph-mon[28050]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:30.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.685+0000 7ff148c00640 1 -- 172.21.15.94:0/2981205663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144104530 msgr2=0x7ff14410adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:30.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.685+0000 7ff148c00640 1 --2- 172.21.15.94:0/2981205663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144104530 0x7ff14410adc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009ca0 tx=0x7ff13c02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.685+0000 7ff148c00640 1 -- 172.21.15.94:0/2981205663 shutdown_connections 2024-04-26T20:07:30.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.685+0000 7ff148c00640 1 --2- 172.21.15.94:0/2981205663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144104530 0x7ff14410adc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.685+0000 7ff148c00640 1 --2- 172.21.15.94:0/2981205663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff144103b70 0x7ff144103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.685+0000 7ff148c00640 1 --2- 172.21.15.94:0/2981205663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff144102980 0x7ff144102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.685+0000 7ff148c00640 1 -- 172.21.15.94:0/2981205663 >> 172.21.15.94:0/2981205663 conn(0x7ff1440fdf10 msgr2=0x7ff144100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:30.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.685+0000 7ff148c00640 1 -- 172.21.15.94:0/2981205663 shutdown_connections 2024-04-26T20:07:30.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.685+0000 7ff148c00640 1 -- 172.21.15.94:0/2981205663 wait complete. 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.686+0000 7ff148c00640 1 Processor -- start 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.686+0000 7ff148c00640 1 -- start start 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.686+0000 7ff148c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff144102980 0x7ff14419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.686+0000 7ff148c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144103b70 0x7ff14419b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.686+0000 7ff148c00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff144104530 0x7ff1441a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.686+0000 7ff148c00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410e500 con 0x7ff144104530 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.686+0000 7ff148c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410e380 con 0x7ff144103b70 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.686+0000 7ff148c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410e680 con 0x7ff144102980 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144103b70 0x7ff14419b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144103b70 0x7ff14419b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:33448/0 (socket says 172.21.15.94:33448) 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13be00640 1 -- 172.21.15.94:0/4171053494 learned_addr learned my addr 172.21.15.94:0/4171053494 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff142800640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff144104530 0x7ff1441a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff141e00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff144102980 0x7ff14419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13be00640 1 -- 172.21.15.94:0/4171053494 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff144102980 msgr2=0x7ff14419adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13be00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff144102980 0x7ff14419adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13be00640 1 -- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff144104530 msgr2=0x7ff1441a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13be00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff144104530 0x7ff1441a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13be00640 1 -- 172.21.15.94:0/4171053494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1441a29c0 con 0x7ff144103b70 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13be00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144103b70 0x7ff14419b2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00ea60 tx=0x7ff12c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.687+0000 7ff13aa00640 1 -- 172.21.15.94:0/4171053494 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c00cde0 con 0x7ff144103b70 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.688+0000 7ff13aa00640 1 -- 172.21.15.94:0/4171053494 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff12c0035f0 con 0x7ff144103b70 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.688+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1441a2cb0 con 0x7ff144103b70 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.688+0000 7ff13aa00640 1 -- 172.21.15.94:0/4171053494 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c010690 con 0x7ff144103b70 2024-04-26T20:07:30.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.688+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1441a31a0 con 0x7ff144103b70 2024-04-26T20:07:30.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.689+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1441a3460 con 0x7ff144103b70 2024-04-26T20:07:30.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.691+0000 7ff13aa00640 1 -- 172.21.15.94:0/4171053494 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff12c0037a0 con 0x7ff144103b70 2024-04-26T20:07:30.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.692+0000 7ff13aa00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff1280a3c30 0x7ff1280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:30.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.692+0000 7ff13aa00640 1 -- 172.21.15.94:0/4171053494 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff12c0d02a0 con 0x7ff144103b70 2024-04-26T20:07:30.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.692+0000 7ff141e00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff1280a3c30 0x7ff1280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:30.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.693+0000 7ff141e00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff1280a3c30 0x7ff1280a60f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff14419c280 tx=0x7ff134005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:30.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.696+0000 7ff13aa00640 1 -- 172.21.15.94:0/4171053494 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff12c09d080 con 0x7ff144103b70 2024-04-26T20:07:30.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.943+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1441a3630 con 0x7ff144103b70 2024-04-26T20:07:30.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.943+0000 7ff13aa00640 1 -- 172.21.15.94:0/4171053494 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7ff12c09c980 con 0x7ff144103b70 2024-04-26T20:07:30.945 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:30.945 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T20:07:30.945 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 3 2024-04-26T20:07:30.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff1280a3c30 msgr2=0x7ff1280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:30.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff1280a3c30 0x7ff1280a60f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff14419c280 tx=0x7ff134005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:30.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144103b70 msgr2=0x7ff14419b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:30.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144103b70 0x7ff14419b2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00ea60 tx=0x7ff12c00ef20 comp rx=0 tx=0).stop 2024-04-26T20:07:30.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 shutdown_connections 2024-04-26T20:07:30.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff1280a3c30 0x7ff1280a60f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff144104530 0x7ff1441a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff144103b70 0x7ff14419b2f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 --2- 172.21.15.94:0/4171053494 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff144102980 0x7ff14419adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:30.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 >> 172.21.15.94:0/4171053494 conn(0x7ff1440fdf10 msgr2=0x7ff144074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:30.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 shutdown_connections 2024-04-26T20:07:30.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:30.946+0000 7ff148c00640 1 -- 172.21.15.94:0/4171053494 wait complete. 2024-04-26T20:07:31.224 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:30 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4171053494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:31.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:30 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4171053494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:31.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:30 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4171053494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:31.488 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:07:31 smithi094 podman[65514]: 2024-04-26 20:07:31.237040337 +0000 UTC m=+0.142807183 container create c8c81c34b8fb405424ea252609b5ee75e7996af673050ac08826ae48b5734a03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T20:07:31.987 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:07:31 smithi094 podman[65514]: 2024-04-26 20:07:31.558653239 +0000 UTC m=+0.464420108 container init c8c81c34b8fb405424ea252609b5ee75e7996af673050ac08826ae48b5734a03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T20:07:31.987 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:07:31 smithi094 podman[65514]: 2024-04-26 20:07:31.568986998 +0000 UTC m=+0.474753874 container start c8c81c34b8fb405424ea252609b5ee75e7996af673050ac08826ae48b5734a03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:07:31.988 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:07:31 smithi094 ceph-mds[65532]: -- 172.21.15.94:0/3511214811 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af9b32ba40 con 0x55af9b2ff000 2024-04-26T20:07:31.988 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:07:31 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j[65528]: starting mds.j at 2024-04-26T20:07:31.988 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:07:31 smithi094 bash[65514]: c8c81c34b8fb405424ea252609b5ee75e7996af673050ac08826ae48b5734a03 2024-04-26T20:07:31.988 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:07:31 smithi094 systemd[1]: Started Ceph mds.j for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:32.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:31 smithi103 ceph-mon[32635]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:32.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:31 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:32.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:31 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:32.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:31 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:32.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:31 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:32.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:31 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:32.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:31 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:32.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:32 smithi204 ceph-mon[32618]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:32.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:32.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:32.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:32.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:32.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:32.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:32 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:32.490 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:31 smithi094 ceph-mon[28050]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:32.491 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:32.491 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:32.491 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:32.491 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:32.491 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:32.491 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:31 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:32.966 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.167+0000 7fbd18400640 1 -- 172.21.15.94:0/2023873307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd1010ab40 msgr2=0x7fbd101113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.167+0000 7fbd18400640 1 --2- 172.21.15.94:0/2023873307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd1010ab40 0x7fbd101113d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009ca0 tx=0x7fbd0802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.167+0000 7fbd18400640 1 -- 172.21.15.94:0/2023873307 shutdown_connections 2024-04-26T20:07:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.167+0000 7fbd18400640 1 --2- 172.21.15.94:0/2023873307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd1010ab40 0x7fbd101113d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.167+0000 7fbd18400640 1 --2- 172.21.15.94:0/2023873307 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd1010a180 0x7fbd1010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.167+0000 7fbd18400640 1 --2- 172.21.15.94:0/2023873307 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd10108f90 0x7fbd101093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.167+0000 7fbd18400640 1 -- 172.21.15.94:0/2023873307 >> 172.21.15.94:0/2023873307 conn(0x7fbd10077f40 msgr2=0x7fbd1007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.167+0000 7fbd18400640 1 -- 172.21.15.94:0/2023873307 shutdown_connections 2024-04-26T20:07:33.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.167+0000 7fbd18400640 1 -- 172.21.15.94:0/2023873307 wait complete. 2024-04-26T20:07:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.168+0000 7fbd18400640 1 Processor -- start 2024-04-26T20:07:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.168+0000 7fbd18400640 1 -- start start 2024-04-26T20:07:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd18400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd10108f90 0x7fbd1019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd18400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd1010a180 0x7fbd1019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd18400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd1010ab40 0x7fbd101a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd18400640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbd10114ad0 con 0x7fbd1010a180 2024-04-26T20:07:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd18400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd10114950 con 0x7fbd1010ab40 2024-04-26T20:07:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd18400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbd10114c50 con 0x7fbd10108f90 2024-04-26T20:07:33.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd15000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd1010a180 0x7fbd1019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd15000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd1010a180 0x7fbd1019f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41108/0 (socket says 172.21.15.94:41108) 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd15000640 1 -- 172.21.15.94:0/2041927165 learned_addr learned my addr 172.21.15.94:0/2041927165 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd15a00640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd10108f90 0x7fbd1019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd16400640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd1010ab40 0x7fbd101a66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd15000640 1 -- 172.21.15.94:0/2041927165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd10108f90 msgr2=0x7fbd1019f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd15000640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd10108f90 0x7fbd1019f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd15000640 1 -- 172.21.15.94:0/2041927165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd1010ab40 msgr2=0x7fbd101a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd15000640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd1010ab40 0x7fbd101a66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.169+0000 7fbd15000640 1 -- 172.21.15.94:0/2041927165 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd101a6d40 con 0x7fbd1010a180 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.170+0000 7fbd15000640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd1010a180 0x7fbd1019f660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00e9f0 tx=0x7fbcfc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.170+0000 7fbd0e000640 1 -- 172.21.15.94:0/2041927165 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00cd80 con 0x7fbd1010a180 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.170+0000 7fbd0e000640 1 -- 172.21.15.94:0/2041927165 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcfc0108e0 con 0x7fbd1010a180 2024-04-26T20:07:33.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.170+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd101a6fd0 con 0x7fbd1010a180 2024-04-26T20:07:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.170+0000 7fbd0e000640 1 -- 172.21.15.94:0/2041927165 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc018980 con 0x7fbd1010a180 2024-04-26T20:07:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.170+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd101a74f0 con 0x7fbd1010a180 2024-04-26T20:07:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.171+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd101102b0 con 0x7fbd1010a180 2024-04-26T20:07:33.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.173+0000 7fbd0e000640 1 -- 172.21.15.94:0/2041927165 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbcfc010430 con 0x7fbd1010a180 2024-04-26T20:07:33.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.175+0000 7fbd0e000640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbcf80a3bb0 0x7fbcf80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:33.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.175+0000 7fbd0e000640 1 -- 172.21.15.94:0/2041927165 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbcfc0d0360 con 0x7fbd1010a180 2024-04-26T20:07:33.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.176+0000 7fbd15a00640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbcf80a3bb0 0x7fbcf80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:33.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.176+0000 7fbd15a00640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbcf80a3bb0 0x7fbcf80a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbd10109fe0 tx=0x7fbd04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:33.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.179+0000 7fbd0e000640 1 -- 172.21.15.94:0/2041927165 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbcfc09d140 con 0x7fbd1010a180 2024-04-26T20:07:33.221 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:33 smithi094 ceph-mon[28050]: Deploying daemon mds.d on smithi094 2024-04-26T20:07:33.221 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:33 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.94:6834/3305420604,v1:172.21.15.94:6835/3305420604] up:boot 2024-04-26T20:07:33.222 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:33 smithi094 ceph-mon[28050]: fsmap 3 up:standby 2024-04-26T20:07:33.222 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:33 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T20:07:33.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:33 smithi103 ceph-mon[32635]: Deploying daemon mds.d on smithi094 2024-04-26T20:07:33.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:33 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.94:6834/3305420604,v1:172.21.15.94:6835/3305420604] up:boot 2024-04-26T20:07:33.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:33 smithi103 ceph-mon[32635]: fsmap 3 up:standby 2024-04-26T20:07:33.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:33 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T20:07:33.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:33 smithi204 ceph-mon[32618]: Deploying daemon mds.d on smithi094 2024-04-26T20:07:33.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:33 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.94:6834/3305420604,v1:172.21.15.94:6835/3305420604] up:boot 2024-04-26T20:07:33.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:33 smithi204 ceph-mon[32618]: fsmap 3 up:standby 2024-04-26T20:07:33.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:33 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T20:07:33.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.425+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd100635d0 con 0x7fbd1010a180 2024-04-26T20:07:33.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.426+0000 7fbd0e000640 1 -- 172.21.15.94:0/2041927165 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fbcfc09ca40 con 0x7fbd1010a180 2024-04-26T20:07:33.428 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:33.428 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T20:07:33.428 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 4 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbcf80a3bb0 msgr2=0x7fbcf80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbcf80a3bb0 0x7fbcf80a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbd10109fe0 tx=0x7fbd04005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd1010a180 msgr2=0x7fbd1019f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd1010a180 0x7fbd1019f660 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00e9f0 tx=0x7fbcfc00eeb0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 shutdown_connections 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbcf80a3bb0 0x7fbcf80a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd1010ab40 0x7fbd101a66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd1010a180 0x7fbd1019f660 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 --2- 172.21.15.94:0/2041927165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd10108f90 0x7fbd1019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.429+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 >> 172.21.15.94:0/2041927165 conn(0x7fbd10077f40 msgr2=0x7fbd1010d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:33.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.430+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 shutdown_connections 2024-04-26T20:07:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:33.430+0000 7fbd18400640 1 -- 172.21.15.94:0/2041927165 wait complete. 2024-04-26T20:07:34.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:34 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2041927165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:34.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:34 smithi094 ceph-mon[28050]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:34.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:34 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2041927165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:34.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:34 smithi103 ceph-mon[32635]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:34.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:34 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2041927165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:34.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:34 smithi204 ceph-mon[32618]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:35.450 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:35.459 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:07:35 smithi094 systemd[1]: Starting Ceph mds.d for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:07:35.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.661+0000 7f33e6800640 1 -- 172.21.15.94:0/2679463522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 msgr2=0x7f33e010abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:35.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.661+0000 7f33e6800640 1 --2- 172.21.15.94:0/2679463522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 0x7f33e010abd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f33d8009ca0 tx=0x7f33d802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.661+0000 7f33e6800640 1 -- 172.21.15.94:0/2679463522 shutdown_connections 2024-04-26T20:07:35.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.661+0000 7f33e6800640 1 --2- 172.21.15.94:0/2679463522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 0x7f33e010abd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.661+0000 7f33e6800640 1 --2- 172.21.15.94:0/2679463522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e0103980 0x7f33e0103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.661+0000 7f33e6800640 1 --2- 172.21.15.94:0/2679463522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e0102790 0x7f33e0102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.661+0000 7f33e6800640 1 -- 172.21.15.94:0/2679463522 >> 172.21.15.94:0/2679463522 conn(0x7f33e00fdd00 msgr2=0x7f33e0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:35.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.661+0000 7f33e6800640 1 -- 172.21.15.94:0/2679463522 shutdown_connections 2024-04-26T20:07:35.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.661+0000 7f33e6800640 1 -- 172.21.15.94:0/2679463522 wait complete. 2024-04-26T20:07:35.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.662+0000 7f33e6800640 1 Processor -- start 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.662+0000 7f33e6800640 1 -- start start 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e6800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e0102790 0x7f33e019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e6800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e0103980 0x7f33e019b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e6800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 0x7f33e01a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e6800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f33e010e2c0 con 0x7f33e0104340 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e6800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33e010e140 con 0x7f33e0103980 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e6800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f33e010e440 con 0x7f33e0102790 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e5e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 0x7f33e01a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e5e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 0x7f33e01a2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41124/0 (socket says 172.21.15.94:41124) 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e0102790 0x7f33e019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e5e00640 1 -- 172.21.15.94:0/2731972852 learned_addr learned my addr 172.21.15.94:0/2731972852 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:35.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.663+0000 7f33e4a00640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e0103980 0x7f33e019b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e5400640 1 -- 172.21.15.94:0/2731972852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e0103980 msgr2=0x7f33e019b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e5400640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e0103980 0x7f33e019b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e5400640 1 -- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 msgr2=0x7f33e01a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e5400640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 0x7f33e01a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e5400640 1 -- 172.21.15.94:0/2731972852 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e01a2800 con 0x7f33e0102790 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e4a00640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e0103980 0x7f33e019b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e5e00640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 0x7f33e01a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e5400640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e0102790 0x7f33e019abe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f33d400e9c0 tx=0x7f33d400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33de000640 1 -- 172.21.15.94:0/2731972852 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d400ccc0 con 0x7f33e0102790 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33de000640 1 -- 172.21.15.94:0/2731972852 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33d40035f0 con 0x7f33e0102790 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e01a2af0 con 0x7f33e0102790 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33de000640 1 -- 172.21.15.94:0/2731972852 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d400ce60 con 0x7f33e0102790 2024-04-26T20:07:35.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.664+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e01a3040 con 0x7f33e0102790 2024-04-26T20:07:35.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.666+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e01a3210 con 0x7f33e0102790 2024-04-26T20:07:35.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.667+0000 7f33de000640 1 -- 172.21.15.94:0/2731972852 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f33d4004000 con 0x7f33e0102790 2024-04-26T20:07:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.669+0000 7f33de000640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33c80a3c30 0x7f33c80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.669+0000 7f33de000640 1 -- 172.21.15.94:0/2731972852 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f33d40d02d0 con 0x7f33e0102790 2024-04-26T20:07:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.669+0000 7f33e4a00640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33c80a3c30 0x7f33c80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:35.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.670+0000 7f33e4a00640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33c80a3c30 0x7f33c80a60f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f33e019c0f0 tx=0x7f33cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:35.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.673+0000 7f33de000640 1 -- 172.21.15.94:0/2731972852 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f33d409d0b0 con 0x7f33e0102790 2024-04-26T20:07:35.737 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:07:35 smithi094 podman[65854]: 2024-04-26 20:07:35.363729299 +0000 UTC m=+0.032974277 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:07:35.738 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:07:35 smithi094 podman[65854]: 2024-04-26 20:07:35.495359066 +0000 UTC m=+0.164604042 container create 394cd22bae4c95b7bd8148e7cd903b03cc55c61d494cf779a0faa971aef101a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:07:35.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.942+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33e01a33e0 con 0x7f33e0102790 2024-04-26T20:07:35.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.943+0000 7f33de000640 1 -- 172.21.15.94:0/2731972852 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f33d409c9b0 con 0x7f33e0102790 2024-04-26T20:07:35.943 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:35.943 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T20:07:35.944 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 4 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33c80a3c30 msgr2=0x7f33c80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33c80a3c30 0x7f33c80a60f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f33e019c0f0 tx=0x7f33cc005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e0102790 msgr2=0x7f33e019abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e0102790 0x7f33e019abe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f33d400e9c0 tx=0x7f33d400ee80 comp rx=0 tx=0).stop 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 shutdown_connections 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33c80a3c30 0x7f33c80a60f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e0104340 0x7f33e01a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e0103980 0x7f33e019b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 --2- 172.21.15.94:0/2731972852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e0102790 0x7f33e019abe0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 >> 172.21.15.94:0/2731972852 conn(0x7f33e00fdd00 msgr2=0x7f33e0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 shutdown_connections 2024-04-26T20:07:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:35.946+0000 7f33e6800640 1 -- 172.21.15.94:0/2731972852 wait complete. 2024-04-26T20:07:36.238 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:07:35 smithi094 podman[65854]: 2024-04-26 20:07:35.807937936 +0000 UTC m=+0.477182950 container init 394cd22bae4c95b7bd8148e7cd903b03cc55c61d494cf779a0faa971aef101a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T20:07:36.238 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:07:35 smithi094 podman[65854]: 2024-04-26 20:07:35.817319839 +0000 UTC m=+0.486564883 container start 394cd22bae4c95b7bd8148e7cd903b03cc55c61d494cf779a0faa971aef101a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T20:07:36.238 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:07:35 smithi094 ceph-mds[65912]: -- 172.21.15.94:0/2990017821 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b48d95fa40 con 0x55b48d932800 2024-04-26T20:07:36.238 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:07:35 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d[65899]: starting mds.d at 2024-04-26T20:07:36.238 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:07:35 smithi094 bash[65854]: 394cd22bae4c95b7bd8148e7cd903b03cc55c61d494cf779a0faa971aef101a4 2024-04-26T20:07:36.238 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:07:35 smithi094 systemd[1]: Started Ceph mds.d for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:36.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:36 smithi204 ceph-mon[32618]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:36.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:36 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2731972852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:36.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:36.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:36.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:36.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:36.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:36.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:36 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:36.955 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:36 smithi103 ceph-mon[32635]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:36.955 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:36 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2731972852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:36.955 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:36.956 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:36.956 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:36.956 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:36.956 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:36.956 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:36 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:36 smithi094 ceph-mon[28050]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:36 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2731972852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:36.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:36.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:36.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:36.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:36.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:36 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:37.877 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:37 smithi103 ceph-mon[32635]: Deploying daemon mds.e on smithi103 2024-04-26T20:07:37.877 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:37 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.94:6836/3734002100,v1:172.21.15.94:6837/3734002100] up:boot 2024-04-26T20:07:37.877 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:37 smithi103 ceph-mon[32635]: fsmap 4 up:standby 2024-04-26T20:07:37.877 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:37 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T20:07:37.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:37 smithi204 ceph-mon[32618]: Deploying daemon mds.e on smithi103 2024-04-26T20:07:37.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:37 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.94:6836/3734002100,v1:172.21.15.94:6837/3734002100] up:boot 2024-04-26T20:07:37.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:37 smithi204 ceph-mon[32618]: fsmap 4 up:standby 2024-04-26T20:07:37.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:37 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T20:07:37.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:37.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:37 smithi094 ceph-mon[28050]: Deploying daemon mds.e on smithi103 2024-04-26T20:07:37.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:37 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.94:6836/3734002100,v1:172.21.15.94:6837/3734002100] up:boot 2024-04-26T20:07:37.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:37 smithi094 ceph-mon[28050]: fsmap 4 up:standby 2024-04-26T20:07:37.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:37 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T20:07:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.178+0000 7fc141a00640 1 -- 172.21.15.94:0/2332036343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc13c076970 msgr2=0x7fc13c074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.178+0000 7fc141a00640 1 --2- 172.21.15.94:0/2332036343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc13c076970 0x7fc13c074dc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009930 tx=0x7fc12c02a420 comp rx=0 tx=0).stop 2024-04-26T20:07:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.179+0000 7fc141a00640 1 -- 172.21.15.94:0/2332036343 shutdown_connections 2024-04-26T20:07:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.179+0000 7fc141a00640 1 --2- 172.21.15.94:0/2332036343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc13c10ed40 0x7fc13c111130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.179+0000 7fc141a00640 1 --2- 172.21.15.94:0/2332036343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc13c075390 0x7fc13c075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.179+0000 7fc141a00640 1 --2- 172.21.15.94:0/2332036343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc13c076970 0x7fc13c074dc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:38.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.179+0000 7fc141a00640 1 -- 172.21.15.94:0/2332036343 >> 172.21.15.94:0/2332036343 conn(0x7fc13c0fdd00 msgr2=0x7fc13c100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:38.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.179+0000 7fc141a00640 1 -- 172.21.15.94:0/2332036343 shutdown_connections 2024-04-26T20:07:38.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.179+0000 7fc141a00640 1 -- 172.21.15.94:0/2332036343 wait complete. 2024-04-26T20:07:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc141a00640 1 Processor -- start 2024-04-26T20:07:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc141a00640 1 -- start start 2024-04-26T20:07:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc141a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc13c075390 0x7fc13c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc141a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc13c076970 0x7fc13c19f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc141a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc13c10ed40 0x7fc13c1a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc141a00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c1148e0 con 0x7fc13c10ed40 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc141a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c114760 con 0x7fc13c075390 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc141a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c114a60 con 0x7fc13c076970 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc13be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc13c075390 0x7fc13c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc13be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc13c075390 0x7fc13c19f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:58028/0 (socket says 172.21.15.94:58028) 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc141000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc13c10ed40 0x7fc13c1a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc13be00640 1 -- 172.21.15.94:0/4271263779 learned_addr learned my addr 172.21.15.94:0/4271263779 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.180+0000 7fc13b400640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc13c076970 0x7fc13c19f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.181+0000 7fc13be00640 1 -- 172.21.15.94:0/4271263779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc13c076970 msgr2=0x7fc13c19f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.181+0000 7fc13be00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc13c076970 0x7fc13c19f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.181+0000 7fc13be00640 1 -- 172.21.15.94:0/4271263779 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc13c10ed40 msgr2=0x7fc13c1a65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.181+0000 7fc13be00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc13c10ed40 0x7fc13c1a65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.181+0000 7fc13be00640 1 -- 172.21.15.94:0/4271263779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc13c1a6c30 con 0x7fc13c075390 2024-04-26T20:07:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.181+0000 7fc13be00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc13c075390 0x7fc13c19f010 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc12c031500 tx=0x7fc12c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.181+0000 7fc138c00640 1 -- 172.21.15.94:0/4271263779 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c03f070 con 0x7fc13c075390 2024-04-26T20:07:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.181+0000 7fc138c00640 1 -- 172.21.15.94:0/4271263779 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc12c031c50 con 0x7fc13c075390 2024-04-26T20:07:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.181+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc13c1a6ec0 con 0x7fc13c075390 2024-04-26T20:07:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.182+0000 7fc138c00640 1 -- 172.21.15.94:0/4271263779 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c035030 con 0x7fc13c075390 2024-04-26T20:07:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.182+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc13c1a72d0 con 0x7fc13c075390 2024-04-26T20:07:38.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.183+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc13c104f10 con 0x7fc13c075390 2024-04-26T20:07:38.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.185+0000 7fc138c00640 1 -- 172.21.15.94:0/4271263779 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc12c042070 con 0x7fc13c075390 2024-04-26T20:07:38.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.186+0000 7fc138c00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc1200a3bb0 0x7fc1200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:38.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.186+0000 7fc138c00640 1 -- 172.21.15.94:0/4271263779 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc12c0f0e80 con 0x7fc13c075390 2024-04-26T20:07:38.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.186+0000 7fc13b400640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc1200a3bb0 0x7fc1200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:38.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.187+0000 7fc13b400640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc1200a3bb0 0x7fc1200a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc13c1a0520 tx=0x7fc128005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:38.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.188+0000 7fc138c00640 1 -- 172.21.15.94:0/4271263779 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc12c0bdc60 con 0x7fc13c075390 2024-04-26T20:07:38.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.433+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc13c02dbf0 con 0x7fc13c075390 2024-04-26T20:07:38.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.434+0000 7fc138c00640 1 -- 172.21.15.94:0/4271263779 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7fc12c0bd560 con 0x7fc13c075390 2024-04-26T20:07:38.435 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:38.435 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T20:07:38.435 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 5 2024-04-26T20:07:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc1200a3bb0 msgr2=0x7fc1200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:38.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc1200a3bb0 0x7fc1200a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc13c1a0520 tx=0x7fc128005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc13c075390 msgr2=0x7fc13c19f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc13c075390 0x7fc13c19f010 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc12c031500 tx=0x7fc12c003c10 comp rx=0 tx=0).stop 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 shutdown_connections 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc1200a3bb0 0x7fc1200a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc13c10ed40 0x7fc13c1a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc13c076970 0x7fc13c19f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 --2- 172.21.15.94:0/4271263779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc13c075390 0x7fc13c19f010 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 >> 172.21.15.94:0/4271263779 conn(0x7fc13c0fdd00 msgr2=0x7fc13c1020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 shutdown_connections 2024-04-26T20:07:38.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:38.437+0000 7fc141a00640 1 -- 172.21.15.94:0/4271263779 wait complete. 2024-04-26T20:07:38.534 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:07:38 smithi103 podman[62585]: 2024-04-26 20:07:38.488281723 +0000 UTC m=+0.141631230 container create e40f6bcc56a925a7975de3c301d5d8cd09fecd5371bd1eeaeb853e100f44ab57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:07:38.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:38 smithi094 ceph-mon[28050]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:38.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:38 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4271263779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:38.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:38 smithi204 ceph-mon[32618]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:38.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:38 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4271263779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:39.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:38 smithi103 ceph-mon[32635]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:39.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:38 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4271263779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:39.035 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:07:38 smithi103 podman[62585]: 2024-04-26 20:07:38.766360012 +0000 UTC m=+0.419709473 container init e40f6bcc56a925a7975de3c301d5d8cd09fecd5371bd1eeaeb853e100f44ab57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-04-26T20:07:39.035 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:07:38 smithi103 podman[62585]: 2024-04-26 20:07:38.769191905 +0000 UTC m=+0.422541375 container start e40f6bcc56a925a7975de3c301d5d8cd09fecd5371bd1eeaeb853e100f44ab57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T20:07:39.035 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:07:38 smithi103 ceph-mds[62605]: -- 172.21.15.103:0/2211055983 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a2ca71fa40 con 0x55a2ca6f2800 2024-04-26T20:07:39.035 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:07:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-e[62601]: starting mds.e at 2024-04-26T20:07:39.035 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:07:38 smithi103 bash[62585]: e40f6bcc56a925a7975de3c301d5d8cd09fecd5371bd1eeaeb853e100f44ab57 2024-04-26T20:07:39.035 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:07:38 smithi103 systemd[1]: Started Ceph mds.e for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: Deploying daemon mds.f on smithi204 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.103:6834/4015617666,v1:172.21.15.103:6835/4015617666] up:boot 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: fsmap 5 up:standby 2024-04-26T20:07:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:39 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T20:07:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: Deploying daemon mds.f on smithi204 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.103:6834/4015617666,v1:172.21.15.103:6835/4015617666] up:boot 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: fsmap 5 up:standby 2024-04-26T20:07:40.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:39 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T20:07:40.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:40.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:40.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:40.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:40.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:40.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:40.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: Deploying daemon mds.f on smithi204 2024-04-26T20:07:40.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:40.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.103:6834/4015617666,v1:172.21.15.103:6835/4015617666] up:boot 2024-04-26T20:07:40.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: fsmap 5 up:standby 2024-04-26T20:07:40.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:39 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T20:07:40.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.671+0000 7f9544a00640 1 -- 172.21.15.94:0/2695771223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9540106560 msgr2=0x7f954010cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.671+0000 7f9544a00640 1 --2- 172.21.15.94:0/2695771223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9540106560 0x7f954010cdf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9538009ca0 tx=0x7f953802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.671+0000 7f9544a00640 1 -- 172.21.15.94:0/2695771223 shutdown_connections 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.671+0000 7f9544a00640 1 --2- 172.21.15.94:0/2695771223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9540106560 0x7f954010cdf0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.671+0000 7f9544a00640 1 --2- 172.21.15.94:0/2695771223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9540105ba0 0x7f9540106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.671+0000 7f9544a00640 1 --2- 172.21.15.94:0/2695771223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95401049b0 0x7f9540104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.671+0000 7f9544a00640 1 -- 172.21.15.94:0/2695771223 >> 172.21.15.94:0/2695771223 conn(0x7f95400fff40 msgr2=0x7f9540102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.671+0000 7f9544a00640 1 -- 172.21.15.94:0/2695771223 shutdown_connections 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.671+0000 7f9544a00640 1 -- 172.21.15.94:0/2695771223 wait complete. 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.672+0000 7f9544a00640 1 Processor -- start 2024-04-26T20:07:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.672+0000 7f9544a00640 1 -- start start 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f9544a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95401049b0 0x7f954019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f9544a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9540105ba0 0x7f954019b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f9544a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9540106560 0x7f95401a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f9544a00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9540110530 con 0x7f95401049b0 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f9544a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95401103b0 con 0x7f9540106560 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f9544a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f95401106b0 con 0x7f9540105ba0 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f953f400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95401049b0 0x7f954019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f953f400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95401049b0 0x7f954019ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41168/0 (socket says 172.21.15.94:41168) 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f953f400640 1 -- 172.21.15.94:0/2164240740 learned_addr learned my addr 172.21.15.94:0/2164240740 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f9536a00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9540105ba0 0x7f954019b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f953fe00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9540106560 0x7f95401a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f953f400640 1 -- 172.21.15.94:0/2164240740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9540105ba0 msgr2=0x7f954019b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f953f400640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9540105ba0 0x7f954019b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.673+0000 7f953f400640 1 -- 172.21.15.94:0/2164240740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9540106560 msgr2=0x7f95401a20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.674+0000 7f953f400640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9540106560 0x7f95401a20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.674+0000 7f953f400640 1 -- 172.21.15.94:0/2164240740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95401a2730 con 0x7f95401049b0 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.674+0000 7f953f400640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95401049b0 0x7f954019ab10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f953000e9c0 tx=0x7f953000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.674+0000 7f953cc00640 1 -- 172.21.15.94:0/2164240740 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953000cde0 con 0x7f95401049b0 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.674+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95401a2a20 con 0x7f95401049b0 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.674+0000 7f953cc00640 1 -- 172.21.15.94:0/2164240740 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95300108e0 con 0x7f95401049b0 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.674+0000 7f953cc00640 1 -- 172.21.15.94:0/2164240740 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95300188d0 con 0x7f95401049b0 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.674+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95401a2fc0 con 0x7f95401049b0 2024-04-26T20:07:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.675+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f954010bc70 con 0x7f95401049b0 2024-04-26T20:07:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.677+0000 7f953cc00640 1 -- 172.21.15.94:0/2164240740 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f953001f020 con 0x7f95401049b0 2024-04-26T20:07:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.678+0000 7f953cc00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95240a3bb0 0x7f95240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.679+0000 7f953cc00640 1 -- 172.21.15.94:0/2164240740 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f95300d0170 con 0x7f95401049b0 2024-04-26T20:07:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.679+0000 7f9536a00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95240a3bb0 0x7f95240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.679+0000 7f9536a00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95240a3bb0 0x7f95240a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f954019c020 tx=0x7f9528005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:40.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.683+0000 7f953cc00640 1 -- 172.21.15.94:0/2164240740 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f953009cf50 con 0x7f95401049b0 2024-04-26T20:07:40.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.928+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f954002dad0 con 0x7f95401049b0 2024-04-26T20:07:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.929+0000 7f953cc00640 1 -- 172.21.15.94:0/2164240740 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7f953009c850 con 0x7f95401049b0 2024-04-26T20:07:40.929 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:40.929 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T20:07:40.929 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 6 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.931+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95240a3bb0 msgr2=0x7f95240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.931+0000 7f9544a00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95240a3bb0 0x7f95240a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f954019c020 tx=0x7f9528005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.931+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95401049b0 msgr2=0x7f954019ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.931+0000 7f9544a00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95401049b0 0x7f954019ab10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f953000e9c0 tx=0x7f953000ee80 comp rx=0 tx=0).stop 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.931+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 shutdown_connections 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.931+0000 7f9544a00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95240a3bb0 0x7f95240a6070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.931+0000 7f9544a00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9540106560 0x7f95401a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.932+0000 7f9544a00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9540105ba0 0x7f954019b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.932+0000 7f9544a00640 1 --2- 172.21.15.94:0/2164240740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95401049b0 0x7f954019ab10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.932+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 >> 172.21.15.94:0/2164240740 conn(0x7f95400fff40 msgr2=0x7f9540108e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:40.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.932+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 shutdown_connections 2024-04-26T20:07:40.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:40.932+0000 7f9544a00640 1 -- 172.21.15.94:0/2164240740 wait complete. 2024-04-26T20:07:41.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:40 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2164240740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:41.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:40 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2164240740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:41.317 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:40 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2164240740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:41.632 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:07:41 smithi204 podman[62865]: 2024-04-26 20:07:41.534623819 +0000 UTC m=+0.115404311 container create 29b9dc616261e478baf328d4ea332d9ae44cfcb8bf4229b34362ec62cffa0783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T20:07:41.896 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:07:41 smithi204 podman[62865]: 2024-04-26 20:07:41.762587162 +0000 UTC m=+0.343367653 container init 29b9dc616261e478baf328d4ea332d9ae44cfcb8bf4229b34362ec62cffa0783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T20:07:41.896 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:07:41 smithi204 podman[62865]: 2024-04-26 20:07:41.765211034 +0000 UTC m=+0.345991533 container start 29b9dc616261e478baf328d4ea332d9ae44cfcb8bf4229b34362ec62cffa0783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-26T20:07:41.896 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:07:41 smithi204 ceph-mds[62884]: -- 172.21.15.204:0/3780958054 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559dcd4fda40 con 0x559dcd4d0800 2024-04-26T20:07:41.896 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:07:41 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f[62880]: starting mds.f at 2024-04-26T20:07:41.897 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:07:41 smithi204 bash[62865]: 29b9dc616261e478baf328d4ea332d9ae44cfcb8bf4229b34362ec62cffa0783 2024-04-26T20:07:42.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:41 smithi094 ceph-mon[28050]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:42.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:41 smithi103 ceph-mon[32635]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:42.381 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:07:41 smithi204 systemd[1]: Started Ceph mds.f for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:42.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:41 smithi204 ceph-mon[32618]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:42.953 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:43.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.172+0000 7f6718000640 1 -- 172.21.15.94:0/1668835219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f671010ab40 msgr2=0x7f67101113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:43.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.172+0000 7f6718000640 1 --2- 172.21.15.94:0/1668835219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f671010ab40 0x7f67101113d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6708009ca0 tx=0x7f670802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.172+0000 7f6718000640 1 -- 172.21.15.94:0/1668835219 shutdown_connections 2024-04-26T20:07:43.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.172+0000 7f6718000640 1 --2- 172.21.15.94:0/1668835219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f671010ab40 0x7f67101113d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.172+0000 7f6718000640 1 --2- 172.21.15.94:0/1668835219 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f671010a180 0x7f671010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.172+0000 7f6718000640 1 --2- 172.21.15.94:0/1668835219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6710108f90 0x7f67101093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.172+0000 7f6718000640 1 -- 172.21.15.94:0/1668835219 >> 172.21.15.94:0/1668835219 conn(0x7f6710077f40 msgr2=0x7f671007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:43.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.172+0000 7f6718000640 1 -- 172.21.15.94:0/1668835219 shutdown_connections 2024-04-26T20:07:43.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.172+0000 7f6718000640 1 -- 172.21.15.94:0/1668835219 wait complete. 2024-04-26T20:07:43.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.173+0000 7f6718000640 1 Processor -- start 2024-04-26T20:07:43.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.173+0000 7f6718000640 1 -- start start 2024-04-26T20:07:43.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6718000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6710108f90 0x7f671019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:43.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6718000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f671010a180 0x7f671019f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:43.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6718000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f671010ab40 0x7f67101a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6718000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6710114ae0 con 0x7f6710108f90 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6718000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6710114960 con 0x7f671010ab40 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6718000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6710114c60 con 0x7f671010a180 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6715600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6710108f90 0x7f671019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6716000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f671010ab40 0x7f67101a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6714c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f671010a180 0x7f671019f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6716000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f671010ab40 0x7f67101a66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:39238/0 (socket says 172.21.15.94:39238) 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6715600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6710108f90 0x7f671019f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42504/0 (socket says 172.21.15.94:42504) 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6716000640 1 -- 172.21.15.94:0/1274299087 learned_addr learned my addr 172.21.15.94:0/1274299087 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6714c00640 1 -- 172.21.15.94:0/1274299087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f671010ab40 msgr2=0x7f67101a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6714c00640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f671010ab40 0x7f67101a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6714c00640 1 -- 172.21.15.94:0/1274299087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6710108f90 msgr2=0x7f671019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6714c00640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6710108f90 0x7f671019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.174+0000 7f6714c00640 1 -- 172.21.15.94:0/1274299087 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67101a6d20 con 0x7f671010a180 2024-04-26T20:07:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.175+0000 7f6714c00640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f671010a180 0x7f671019f640 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66fc00e9f0 tx=0x7f66fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.175+0000 7f670e000640 1 -- 172.21.15.94:0/1274299087 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc00ccc0 con 0x7f671010a180 2024-04-26T20:07:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.175+0000 7f670e000640 1 -- 172.21.15.94:0/1274299087 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66fc0108e0 con 0x7f671010a180 2024-04-26T20:07:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.175+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67101a6fb0 con 0x7f671010a180 2024-04-26T20:07:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.175+0000 7f670e000640 1 -- 172.21.15.94:0/1274299087 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc018880 con 0x7f671010a180 2024-04-26T20:07:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.175+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67101a74d0 con 0x7f671010a180 2024-04-26T20:07:43.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.176+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67101102b0 con 0x7f671010a180 2024-04-26T20:07:43.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.178+0000 7f670e000640 1 -- 172.21.15.94:0/1274299087 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f66fc010430 con 0x7f671010a180 2024-04-26T20:07:43.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.179+0000 7f670e000640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f66f80a3bb0 0x7f66f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:43.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.180+0000 7f670e000640 1 -- 172.21.15.94:0/1274299087 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f66fc0d02d0 con 0x7f671010a180 2024-04-26T20:07:43.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.180+0000 7f6715600640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f66f80a3bb0 0x7f66f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:43.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.180+0000 7f6715600640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f66f80a3bb0 0x7f66f80a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f67040095f0 tx=0x7f6704005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:43.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.184+0000 7f670e000640 1 -- 172.21.15.94:0/1274299087 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f66fc09ca60 con 0x7f671010a180 2024-04-26T20:07:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.204:6834/4163398350,v1:172.21.15.204:6835/4163398350] up:boot 2024-04-26T20:07:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: fsmap 6 up:standby 2024-04-26T20:07:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:07:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:43.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:43.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:43.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:42 smithi094 ceph-mon[28050]: Deploying daemon mds.c on smithi204 2024-04-26T20:07:43.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:43.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:43.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.204:6834/4163398350,v1:172.21.15.204:6835/4163398350] up:boot 2024-04-26T20:07:43.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: fsmap 6 up:standby 2024-04-26T20:07:43.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:07:43.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:43.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:43.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:43.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:43.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:42 smithi103 ceph-mon[32635]: Deploying daemon mds.c on smithi204 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.204:6834/4163398350,v1:172.21.15.204:6835/4163398350] up:boot 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: fsmap 6 up:standby 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:43.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:42 smithi204 ceph-mon[32618]: Deploying daemon mds.c on smithi204 2024-04-26T20:07:43.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.424+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67100635d0 con 0x7f671010a180 2024-04-26T20:07:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.425+0000 7f670e000640 1 -- 172.21.15.94:0/1274299087 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f66fc0a0d60 con 0x7f671010a180 2024-04-26T20:07:43.426 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:43.426 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-26T20:07:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 7 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f66f80a3bb0 msgr2=0x7f66f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f66f80a3bb0 0x7f66f80a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f67040095f0 tx=0x7f6704005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f671010a180 msgr2=0x7f671019f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f671010a180 0x7f671019f640 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66fc00e9f0 tx=0x7f66fc00eeb0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 shutdown_connections 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f66f80a3bb0 0x7f66f80a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f671010ab40 0x7f67101a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f671010a180 0x7f671019f640 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 --2- 172.21.15.94:0/1274299087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6710108f90 0x7f671019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.428+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 >> 172.21.15.94:0/1274299087 conn(0x7f6710077f40 msgr2=0x7f671010d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.429+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 shutdown_connections 2024-04-26T20:07:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:43.429+0000 7f6718000640 1 -- 172.21.15.94:0/1274299087 wait complete. 2024-04-26T20:07:44.057 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:43 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1274299087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:44.057 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:43 smithi204 ceph-mon[32618]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:44.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:43 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1274299087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:44.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:43 smithi094 ceph-mon[28050]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:44.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:43 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1274299087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:44.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:43 smithi103 ceph-mon[32635]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:44.382 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:07:44 smithi204 systemd[1]: Starting Ceph mds.c for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:07:44.719 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:07:44 smithi204 podman[63170]: 2024-04-26 20:07:44.363570044 +0000 UTC m=+0.018324697 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:07:44.719 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:07:44 smithi204 podman[63170]: 2024-04-26 20:07:44.483132744 +0000 UTC m=+0.137887431 container create 941a731dbf396b39e09599b68541e8b4c1ac282b46b5fbd830f87bfcd061c052 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True) 2024-04-26T20:07:44.719 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:07:44 smithi204 podman[63170]: 2024-04-26 20:07:44.719158522 +0000 UTC m=+0.373913164 container init 941a731dbf396b39e09599b68541e8b4c1ac282b46b5fbd830f87bfcd061c052 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T20:07:45.131 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:07:44 smithi204 podman[63170]: 2024-04-26 20:07:44.721770143 +0000 UTC m=+0.376524793 container start 941a731dbf396b39e09599b68541e8b4c1ac282b46b5fbd830f87bfcd061c052 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T20:07:45.132 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:07:44 smithi204 ceph-mds[63190]: -- 172.21.15.204:0/2977853929 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55837d511a40 con 0x55837d4e4c00 2024-04-26T20:07:45.132 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:07:44 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c[63186]: starting mds.c at 2024-04-26T20:07:45.132 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:07:44 smithi204 bash[63170]: 941a731dbf396b39e09599b68541e8b4c1ac282b46b5fbd830f87bfcd061c052 2024-04-26T20:07:45.132 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:07:44 smithi204 systemd[1]: Started Ceph mds.c for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:45.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:45.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.644+0000 7f2e05a00640 1 -- 172.21.15.94:0/3754532690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e00105d20 msgr2=0x7f2e00108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:45.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.644+0000 7f2e05a00640 1 --2- 172.21.15.94:0/3754532690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e00105d20 0x7f2e00108110 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2de8009900 tx=0x7f2de802a410 comp rx=0 tx=0).stop 2024-04-26T20:07:45.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.644+0000 7f2e05a00640 1 -- 172.21.15.94:0/3754532690 shutdown_connections 2024-04-26T20:07:45.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.644+0000 7f2e05a00640 1 --2- 172.21.15.94:0/3754532690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e00108650 0x7f2e0010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.644+0000 7f2e05a00640 1 --2- 172.21.15.94:0/3754532690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e00105d20 0x7f2e00108110 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.644+0000 7f2e05a00640 1 --2- 172.21.15.94:0/3754532690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e001033f0 0x7f2e001057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.644+0000 7f2e05a00640 1 -- 172.21.15.94:0/3754532690 >> 172.21.15.94:0/3754532690 conn(0x7f2e000fcde0 msgr2=0x7f2e000ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:45.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.644+0000 7f2e05a00640 1 -- 172.21.15.94:0/3754532690 shutdown_connections 2024-04-26T20:07:45.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.644+0000 7f2e05a00640 1 -- 172.21.15.94:0/3754532690 wait complete. 2024-04-26T20:07:45.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2e05a00640 1 Processor -- start 2024-04-26T20:07:45.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2e05a00640 1 -- start start 2024-04-26T20:07:45.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2e05a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e001033f0 0x7f2e0019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:45.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2e05a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e00105d20 0x7f2e0019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:45.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2e05a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e00108650 0x7f2e001a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2e05a00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0010e4e0 con 0x7f2e001033f0 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2e05a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0010e360 con 0x7f2e00108650 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2e05a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0010e660 con 0x7f2e00105d20 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2dfea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e001033f0 0x7f2e0019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.645+0000 7f2dfea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e001033f0 0x7f2e0019ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42530/0 (socket says 172.21.15.94:42530) 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2dfea00640 1 -- 172.21.15.94:0/2500832191 learned_addr learned my addr 172.21.15.94:0/2500832191 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2df6000640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e00105d20 0x7f2e0019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2dff400640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e00108650 0x7f2e001a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2dfea00640 1 -- 172.21.15.94:0/2500832191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e00105d20 msgr2=0x7f2e0019b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2dfea00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e00105d20 0x7f2e0019b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2dfea00640 1 -- 172.21.15.94:0/2500832191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e00108650 msgr2=0x7f2e001a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2dfea00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e00108650 0x7f2e001a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2dfea00640 1 -- 172.21.15.94:0/2500832191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e001a2a20 con 0x7f2e001033f0 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2dfea00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e001033f0 0x7f2e0019ae00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2df000ebc0 tx=0x7f2df000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:45.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2df7e00640 1 -- 172.21.15.94:0/2500832191 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0010430 con 0x7f2e001033f0 2024-04-26T20:07:45.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2df7e00640 1 -- 172.21.15.94:0/2500832191 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2df0010a80 con 0x7f2e001033f0 2024-04-26T20:07:45.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e001a2d10 con 0x7f2e001033f0 2024-04-26T20:07:45.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.646+0000 7f2df7e00640 1 -- 172.21.15.94:0/2500832191 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0018a50 con 0x7f2e001033f0 2024-04-26T20:07:45.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.647+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e001a31d0 con 0x7f2e001033f0 2024-04-26T20:07:45.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.648+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e0010a1a0 con 0x7f2e001033f0 2024-04-26T20:07:45.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.650+0000 7f2df7e00640 1 -- 172.21.15.94:0/2500832191 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2df0010c30 con 0x7f2e001033f0 2024-04-26T20:07:45.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.651+0000 7f2df7e00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2de40a3c30 0x7f2de40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:45.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.651+0000 7f2df7e00640 1 -- 172.21.15.94:0/2500832191 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2df00d0620 con 0x7f2e001033f0 2024-04-26T20:07:45.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.651+0000 7f2df6000640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2de40a3c30 0x7f2de40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:45.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.652+0000 7f2df6000640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2de40a3c30 0x7f2de40a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2de8005ab0 tx=0x7f2de8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:45.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.656+0000 7f2df7e00640 1 -- 172.21.15.94:0/2500832191 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2df009d400 con 0x7f2e001033f0 2024-04-26T20:07:45.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.897+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e0002dad0 con 0x7f2e001033f0 2024-04-26T20:07:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.898+0000 7f2df7e00640 1 -- 172.21.15.94:0/2500832191 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f2df009cd00 con 0x7f2e001033f0 2024-04-26T20:07:45.899 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:45.900 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T20:07:45.900 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 8 2024-04-26T20:07:45.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.900+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2de40a3c30 msgr2=0x7f2de40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:45.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.900+0000 7f2e05a00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2de40a3c30 0x7f2de40a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2de8005ab0 tx=0x7f2de8005a20 comp rx=0 tx=0).stop 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.900+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e001033f0 msgr2=0x7f2e0019ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.900+0000 7f2e05a00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e001033f0 0x7f2e0019ae00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2df000ebc0 tx=0x7f2df000c2d0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.901+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 shutdown_connections 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.901+0000 7f2e05a00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2de40a3c30 0x7f2de40a60f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.901+0000 7f2e05a00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e00108650 0x7f2e001a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.901+0000 7f2e05a00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e00105d20 0x7f2e0019b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.901+0000 7f2e05a00640 1 --2- 172.21.15.94:0/2500832191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2e001033f0 0x7f2e0019ae00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.901+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 >> 172.21.15.94:0/2500832191 conn(0x7f2e000fcde0 msgr2=0x7f2e00102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.901+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 shutdown_connections 2024-04-26T20:07:45.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:45.901+0000 7f2e05a00640 1 -- 172.21.15.94:0/2500832191 wait complete. 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: Deploying daemon mds.h on smithi103 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.204:6836/820565196,v1:172.21.15.204:6837/820565196] up:boot 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: fsmap 7 up:standby 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:07:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:45 smithi204 ceph-mon[32618]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:46.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:46.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:46.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:46.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:46.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:46.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:46.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: Deploying daemon mds.h on smithi103 2024-04-26T20:07:46.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.204:6836/820565196,v1:172.21.15.204:6837/820565196] up:boot 2024-04-26T20:07:46.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: fsmap 7 up:standby 2024-04-26T20:07:46.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:07:46.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:45 smithi094 ceph-mon[28050]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: Deploying daemon mds.h on smithi103 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.204:6836/820565196,v1:172.21.15.204:6837/820565196] up:boot 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: fsmap 7 up:standby 2024-04-26T20:07:46.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:07:46.267 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:45 smithi103 ceph-mon[32635]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:47.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:46 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2500832191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:47.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:46 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2500832191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:47.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:46 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2500832191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:47.266 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:07:47 smithi103 systemd[1]: Starting Ceph mds.h for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:07:47.530 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:07:47 smithi103 podman[62904]: 2024-04-26 20:07:47.153799109 +0000 UTC m=+0.017872978 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:07:47.547 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:07:47 smithi103 podman[62904]: 2024-04-26 20:07:47.277352957 +0000 UTC m=+0.141426854 container create 11411d6eab466099cf2335b2a5dd11a4afbeef392e52353b14dbacfdf6a504bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:07:47.555 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:07:47 smithi103 podman[62904]: 2024-04-26 20:07:47.530405277 +0000 UTC m=+0.394479132 container init 11411d6eab466099cf2335b2a5dd11a4afbeef392e52353b14dbacfdf6a504bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:07:47.784 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:07:47 smithi103 podman[62904]: 2024-04-26 20:07:47.53307981 +0000 UTC m=+0.397153675 container start 11411d6eab466099cf2335b2a5dd11a4afbeef392e52353b14dbacfdf6a504bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:07:47.808 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:07:47 smithi103 ceph-mds[62923]: -- 172.21.15.103:0/30345406 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5597d60cba40 con 0x5597d609e800 2024-04-26T20:07:47.818 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:07:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h[62919]: starting mds.h at 2024-04-26T20:07:47.827 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:07:47 smithi103 bash[62904]: 11411d6eab466099cf2335b2a5dd11a4afbeef392e52353b14dbacfdf6a504bf 2024-04-26T20:07:47.836 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:07:47 smithi103 systemd[1]: Started Ceph mds.h for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:47.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:48.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.140+0000 7fa932200640 1 -- 172.21.15.94:0/446557627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa92c106740 msgr2=0x7fa92c10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:48.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.140+0000 7fa932200640 1 --2- 172.21.15.94:0/446557627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa92c106740 0x7fa92c10cfd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa924009be0 tx=0x7fa92402a6c0 comp rx=0 tx=0).stop 2024-04-26T20:07:48.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.140+0000 7fa932200640 1 -- 172.21.15.94:0/446557627 shutdown_connections 2024-04-26T20:07:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.140+0000 7fa932200640 1 --2- 172.21.15.94:0/446557627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa92c106740 0x7fa92c10cfd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:48.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.140+0000 7fa932200640 1 --2- 172.21.15.94:0/446557627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa92c105d80 0x7fa92c106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:48.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.140+0000 7fa932200640 1 --2- 172.21.15.94:0/446557627 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa92c104b90 0x7fa92c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:48.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.140+0000 7fa932200640 1 -- 172.21.15.94:0/446557627 >> 172.21.15.94:0/446557627 conn(0x7fa92c100120 msgr2=0x7fa92c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:48.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.140+0000 7fa932200640 1 -- 172.21.15.94:0/446557627 shutdown_connections 2024-04-26T20:07:48.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.140+0000 7fa932200640 1 -- 172.21.15.94:0/446557627 wait complete. 2024-04-26T20:07:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.141+0000 7fa932200640 1 Processor -- start 2024-04-26T20:07:48.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.141+0000 7fa932200640 1 -- start start 2024-04-26T20:07:48.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa932200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa92c104b90 0x7fa92c19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:48.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa932200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa92c105d80 0x7fa92c19b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:48.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa932200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa92c106740 0x7fa92c1a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:48.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa932200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa92c1106a0 con 0x7fa92c104b90 2024-04-26T20:07:48.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa932200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa92c110520 con 0x7fa92c105d80 2024-04-26T20:07:48.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa932200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa92c110820 con 0x7fa92c106740 2024-04-26T20:07:48.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa92b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa92c104b90 0x7fa92c19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:48.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa92b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa92c104b90 0x7fa92c19ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42548/0 (socket says 172.21.15.94:42548) 2024-04-26T20:07:48.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa92b400640 1 -- 172.21.15.94:0/2350104186 learned_addr learned my addr 172.21.15.94:0/2350104186 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:48.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa92be00640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa92c106740 0x7fa92c1a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:48.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa922a00640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa92c105d80 0x7fa92c19b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:48.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa92b400640 1 -- 172.21.15.94:0/2350104186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa92c106740 msgr2=0x7fa92c1a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:48.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa92b400640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa92c106740 0x7fa92c1a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:48.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa92b400640 1 -- 172.21.15.94:0/2350104186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa92c105d80 msgr2=0x7fa92c19b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:48.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa92b400640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa92c105d80 0x7fa92c19b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:48.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.142+0000 7fa92b400640 1 -- 172.21.15.94:0/2350104186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92c1a2940 con 0x7fa92c104b90 2024-04-26T20:07:48.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.143+0000 7fa92b400640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa92c104b90 0x7fa92c19ad30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00e9f0 tx=0x7fa91c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.143+0000 7fa928c00640 1 -- 172.21.15.94:0/2350104186 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c00ccc0 con 0x7fa92c104b90 2024-04-26T20:07:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.143+0000 7fa928c00640 1 -- 172.21.15.94:0/2350104186 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa91c0035f0 con 0x7fa92c104b90 2024-04-26T20:07:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.143+0000 7fa928c00640 1 -- 172.21.15.94:0/2350104186 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c00ce60 con 0x7fa92c104b90 2024-04-26T20:07:48.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.143+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa92c1a2c30 con 0x7fa92c104b90 2024-04-26T20:07:48.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.143+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa92c1a3180 con 0x7fa92c104b90 2024-04-26T20:07:48.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.144+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa92c10beb0 con 0x7fa92c104b90 2024-04-26T20:07:48.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.146+0000 7fa928c00640 1 -- 172.21.15.94:0/2350104186 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa91c004000 con 0x7fa92c104b90 2024-04-26T20:07:48.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.147+0000 7fa928c00640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9100a3bb0 0x7fa9100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:48.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.148+0000 7fa928c00640 1 -- 172.21.15.94:0/2350104186 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa91c0d0270 con 0x7fa92c104b90 2024-04-26T20:07:48.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.148+0000 7fa922a00640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9100a3bb0 0x7fa9100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:48.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.148+0000 7fa922a00640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9100a3bb0 0x7fa9100a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa9140095f0 tx=0x7fa914005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:48.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.152+0000 7fa928c00640 1 -- 172.21.15.94:0/2350104186 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa91c09d050 con 0x7fa92c104b90 2024-04-26T20:07:48.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.393+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa92c02dad0 con 0x7fa92c104b90 2024-04-26T20:07:48.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.394+0000 7fa928c00640 1 -- 172.21.15.94:0/2350104186 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7180 (secure 0 0 0) 0x7fa91c09c950 con 0x7fa92c104b90 2024-04-26T20:07:48.846 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:48.854 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T20:07:48.907 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-04-26T20:07:48.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.396+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9100a3bb0 msgr2=0x7fa9100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:48.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.396+0000 7fa932200640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9100a3bb0 0x7fa9100a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa9140095f0 tx=0x7fa914005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:49.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa92c104b90 msgr2=0x7fa92c19ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:49.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa92c104b90 0x7fa92c19ad30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00e9f0 tx=0x7fa91c00eeb0 comp rx=0 tx=0).stop 2024-04-26T20:07:49.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 shutdown_connections 2024-04-26T20:07:49.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9100a3bb0 0x7fa9100a6070 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:49.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa92c106740 0x7fa92c1a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa92c105d80 0x7fa92c19b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:49.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 --2- 172.21.15.94:0/2350104186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa92c104b90 0x7fa92c19ad30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:49.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 >> 172.21.15.94:0/2350104186 conn(0x7fa92c100120 msgr2=0x7fa92c109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:49.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 shutdown_connections 2024-04-26T20:07:49.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:48.397+0000 7fa932200640 1 -- 172.21.15.94:0/2350104186 wait complete. 2024-04-26T20:07:49.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:49.192 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:49.201 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:49.208 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:49.250 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:49.259 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:49.268 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:49.276 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.103:6836/3697356689,v1:172.21.15.103:6837/3697356689] up:boot 2024-04-26T20:07:49.284 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: fsmap 8 up:standby 2024-04-26T20:07:49.294 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:07:49.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:48 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2350104186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:49.313 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:49.324 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:49.336 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:49.343 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:49.401 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:49.459 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:49.469 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:49.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.103:6836/3697356689,v1:172.21.15.103:6837/3697356689] up:boot 2024-04-26T20:07:49.495 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: fsmap 8 up:standby 2024-04-26T20:07:49.503 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:07:49.542 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:48 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2350104186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:49.550 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:49.557 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:49.563 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:49.576 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:49.582 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:49.588 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:49.595 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:49.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.103:6836/3697356689,v1:172.21.15.103:6837/3697356689] up:boot 2024-04-26T20:07:49.633 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: fsmap 8 up:standby 2024-04-26T20:07:49.639 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:07:49.707 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:48 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2350104186' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:49.969 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:49 smithi204 ceph-mon[32618]: Deploying daemon mds.i on smithi204 2024-04-26T20:07:50.023 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:07:49 smithi204 systemd[1]: Starting Ceph mds.i for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:07:50.033 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:49 smithi094 ceph-mon[28050]: Deploying daemon mds.i on smithi204 2024-04-26T20:07:50.042 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:49 smithi103 ceph-mon[32635]: Deploying daemon mds.i on smithi204 2024-04-26T20:07:50.338 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:07:50 smithi204 podman[63481]: 2024-04-26 20:07:50.089333323 +0000 UTC m=+0.018267021 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:07:50.354 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:07:50 smithi204 podman[63481]: 2024-04-26 20:07:50.21461598 +0000 UTC m=+0.143549661 container create ba280396131d3f17c199f7b58db94297df8c1f4ad8814f12d5d17f3f9502980f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-26T20:07:50.632 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:07:50 smithi204 podman[63481]: 2024-04-26 20:07:50.451183442 +0000 UTC m=+0.380117123 container init ba280396131d3f17c199f7b58db94297df8c1f4ad8814f12d5d17f3f9502980f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T20:07:50.646 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:07:50 smithi204 podman[63481]: 2024-04-26 20:07:50.453788132 +0000 UTC m=+0.382721813 container start ba280396131d3f17c199f7b58db94297df8c1f4ad8814f12d5d17f3f9502980f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T20:07:50.655 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:07:50 smithi204 ceph-mds[63500]: -- 172.21.15.204:0/4226432823 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619ef763a40 con 0x5619ef737000 2024-04-26T20:07:50.661 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:07:50 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i[63496]: starting mds.i at 2024-04-26T20:07:50.691 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:07:50 smithi204 bash[63481]: ba280396131d3f17c199f7b58db94297df8c1f4ad8814f12d5d17f3f9502980f 2024-04-26T20:07:50.698 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:07:50 smithi204 systemd[1]: Started Ceph mds.i for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:50.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:50 smithi094 ceph-mon[28050]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:51.008 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:51.014 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:51.022 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:51.113 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:51.128 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:51.141 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:50 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:51.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:50 smithi204 ceph-mon[32618]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:51.168 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:51.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:51.196 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:51.258 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:51.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:51.285 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:50 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:51.298 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:50 smithi103 ceph-mon[32635]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:51.310 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:51.322 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:51.333 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:51.426 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:51.436 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:51.442 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:50 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:51.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:51.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.723+0000 7f3291000640 1 -- 172.21.15.94:0/1721407063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f328c104b90 msgr2=0x7f328c104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:51.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.723+0000 7f3291000640 1 --2- 172.21.15.94:0/1721407063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f328c104b90 0x7f328c104fb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f327c009900 tx=0x7f327c02a410 comp rx=0 tx=0).stop 2024-04-26T20:07:51.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.723+0000 7f3291000640 1 -- 172.21.15.94:0/1721407063 shutdown_connections 2024-04-26T20:07:51.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.723+0000 7f3291000640 1 --2- 172.21.15.94:0/1721407063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f328c106740 0x7f328c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:51.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.723+0000 7f3291000640 1 --2- 172.21.15.94:0/1721407063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c105d80 0x7f328c106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:51.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.723+0000 7f3291000640 1 --2- 172.21.15.94:0/1721407063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f328c104b90 0x7f328c104fb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:51.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.723+0000 7f3291000640 1 -- 172.21.15.94:0/1721407063 >> 172.21.15.94:0/1721407063 conn(0x7f328c100120 msgr2=0x7f328c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:51.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.723+0000 7f3291000640 1 -- 172.21.15.94:0/1721407063 shutdown_connections 2024-04-26T20:07:51.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.723+0000 7f3291000640 1 -- 172.21.15.94:0/1721407063 wait complete. 2024-04-26T20:07:51.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:51 smithi204 ceph-mon[32618]: Deploying daemon mds.g on smithi094 2024-04-26T20:07:51.892 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:51 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] up:boot 2024-04-26T20:07:51.903 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:51 smithi204 ceph-mon[32618]: fsmap 9 up:standby 2024-04-26T20:07:51.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:51 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:07:51.966 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:51 smithi204 ceph-mon[32618]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:51.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.724+0000 7f3291000640 1 Processor -- start 2024-04-26T20:07:51.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.724+0000 7f3291000640 1 -- start start 2024-04-26T20:07:52.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.724+0000 7f3291000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f328c104b90 0x7f328c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:52.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.724+0000 7f3291000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c105d80 0x7f328c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:52.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.724+0000 7f3291000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f328c106740 0x7f328c1a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:52.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.724+0000 7f3291000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f328c1106e0 con 0x7f328c104b90 2024-04-26T20:07:52.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.724+0000 7f3291000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f328c110560 con 0x7f328c106740 2024-04-26T20:07:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.724+0000 7f3291000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f328c110860 con 0x7f328c105d80 2024-04-26T20:07:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f328c104b90 0x7f328c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:52.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c105d80 0x7f328c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c105d80 0x7f328c19b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:34900/0 (socket says 172.21.15.94:34900) 2024-04-26T20:07:52.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289400640 1 -- 172.21.15.94:0/2052372063 learned_addr learned my addr 172.21.15.94:0/2052372063 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:52.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f328a800640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f328c106740 0x7f328c1a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:52.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289400640 1 -- 172.21.15.94:0/2052372063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f328c106740 msgr2=0x7f328c1a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:52.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289400640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f328c106740 0x7f328c1a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:52.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289400640 1 -- 172.21.15.94:0/2052372063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f328c104b90 msgr2=0x7f328c19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:52.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289400640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f328c104b90 0x7f328c19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:52.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289400640 1 -- 172.21.15.94:0/2052372063 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328c1a29d0 con 0x7f328c105d80 2024-04-26T20:07:52.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f328a800640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f328c106740 0x7f328c1a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:07:52.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3289400640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c105d80 0x7f328c19b300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f327800e9f0 tx=0x7f327800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.725+0000 7f3282a00640 1 -- 172.21.15.94:0/2052372063 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327800cd80 con 0x7f328c105d80 2024-04-26T20:07:52.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.726+0000 7f3282a00640 1 -- 172.21.15.94:0/2052372063 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32780108e0 con 0x7f328c105d80 2024-04-26T20:07:52.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.726+0000 7f3282a00640 1 -- 172.21.15.94:0/2052372063 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278018980 con 0x7f328c105d80 2024-04-26T20:07:52.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.726+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328c1a2cc0 con 0x7f328c105d80 2024-04-26T20:07:52.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.726+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f328c1a3190 con 0x7f328c105d80 2024-04-26T20:07:52.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.727+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f328c10beb0 con 0x7f328c105d80 2024-04-26T20:07:52.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.728+0000 7f3282a00640 1 -- 172.21.15.94:0/2052372063 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3278010430 con 0x7f328c105d80 2024-04-26T20:07:52.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.730+0000 7f3282a00640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32700a3bb0 0x7f32700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.730+0000 7f3282a00640 1 -- 172.21.15.94:0/2052372063 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f32780d0410 con 0x7f328c105d80 2024-04-26T20:07:52.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.730+0000 7f3289e00640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32700a3bb0 0x7f32700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:52.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.730+0000 7f3289e00640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32700a3bb0 0x7f32700a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f327c005ab0 tx=0x7f327c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:52.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.734+0000 7f3282a00640 1 -- 172.21.15.94:0/2052372063 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f327809d1f0 con 0x7f328c105d80 2024-04-26T20:07:52.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:51 smithi094 ceph-mon[28050]: Deploying daemon mds.g on smithi094 2024-04-26T20:07:52.772 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:51 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] up:boot 2024-04-26T20:07:52.782 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:51 smithi094 ceph-mon[28050]: fsmap 9 up:standby 2024-04-26T20:07:52.794 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:51 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:07:52.878 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:51 smithi094 ceph-mon[28050]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:52.929 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:51 smithi103 ceph-mon[32635]: Deploying daemon mds.g on smithi094 2024-04-26T20:07:53.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:51 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] up:boot 2024-04-26T20:07:53.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:51 smithi103 ceph-mon[32635]: fsmap 9 up:standby 2024-04-26T20:07:53.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:51 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:07:53.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:51 smithi103 ceph-mon[32635]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:53.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.979+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f328c02dad0 con 0x7f328c105d80 2024-04-26T20:07:53.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.980+0000 7f3282a00640 1 -- 172.21.15.94:0/2052372063 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7f327809de20 con 0x7f328c105d80 2024-04-26T20:07:53.128 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:53.144 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T20:07:53.159 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 10 2024-04-26T20:07:53.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32700a3bb0 msgr2=0x7f32700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:53.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32700a3bb0 0x7f32700a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f327c005ab0 tx=0x7f327c005a20 comp rx=0 tx=0).stop 2024-04-26T20:07:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c105d80 msgr2=0x7f328c19b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:53.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c105d80 0x7f328c19b300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f327800e9f0 tx=0x7f327800eeb0 comp rx=0 tx=0).stop 2024-04-26T20:07:53.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 shutdown_connections 2024-04-26T20:07:53.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32700a3bb0 0x7f32700a6070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:53.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f328c106740 0x7f328c1a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:53.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c105d80 0x7f328c19b300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:53.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 --2- 172.21.15.94:0/2052372063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f328c104b90 0x7f328c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 >> 172.21.15.94:0/2052372063 conn(0x7f328c100120 msgr2=0x7f328c109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:53.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.983+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 shutdown_connections 2024-04-26T20:07:53.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:51.984+0000 7f3291000640 1 -- 172.21.15.94:0/2052372063 wait complete. 2024-04-26T20:07:53.567 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:52 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2052372063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:53.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:52 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2052372063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:53.630 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:52 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2052372063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:53.842 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:53 smithi094 ceph-mon[28050]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:54.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:53 smithi204 ceph-mon[32618]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:54.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:53 smithi103 ceph-mon[32635]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:54.205 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:07:53 smithi094 systemd[1]: Starting Ceph mds.g for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:07:54.461 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:07:54 smithi094 podman[66461]: 2024-04-26 20:07:54.108213316 +0000 UTC m=+0.035813661 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:07:54.526 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:07:54 smithi094 podman[66461]: 2024-04-26 20:07:54.269296333 +0000 UTC m=+0.196896718 container create 6bc36881c32d23d1322a2e537ac3d1aeaeb7ff2e80c428a0d211db7977aad451 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T20:07:54.737 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:07:54 smithi094 podman[66461]: 2024-04-26 20:07:54.572773422 +0000 UTC m=+0.500373811 container init 6bc36881c32d23d1322a2e537ac3d1aeaeb7ff2e80c428a0d211db7977aad451 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T20:07:54.759 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:07:54 smithi094 podman[66461]: 2024-04-26 20:07:54.583416581 +0000 UTC m=+0.511016953 container start 6bc36881c32d23d1322a2e537ac3d1aeaeb7ff2e80c428a0d211db7977aad451 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:07:54.868 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:07:54 smithi094 ceph-mds[66481]: -- 172.21.15.94:0/2154810813 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fac5707a40 con 0x55fac56da800 2024-04-26T20:07:54.882 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:07:54 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g[66477]: starting mds.g at 2024-04-26T20:07:54.895 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:07:54 smithi094 bash[66461]: 6bc36881c32d23d1322a2e537ac3d1aeaeb7ff2e80c428a0d211db7977aad451 2024-04-26T20:07:54.908 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:07:54 smithi094 systemd[1]: Started Ceph mds.g for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:55.567 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:07:55.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.781+0000 7f120e000640 1 -- 172.21.15.94:0/2382567288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208105d80 msgr2=0x7f1208106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:55.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.781+0000 7f120e000640 1 --2- 172.21.15.94:0/2382567288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208105d80 0x7f1208106200 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f11f0009900 tx=0x7f11f002a410 comp rx=0 tx=0).stop 2024-04-26T20:07:55.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.782+0000 7f120e000640 1 -- 172.21.15.94:0/2382567288 shutdown_connections 2024-04-26T20:07:55.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.782+0000 7f120e000640 1 --2- 172.21.15.94:0/2382567288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1208106740 0x7f120810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:55.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.782+0000 7f120e000640 1 --2- 172.21.15.94:0/2382567288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208105d80 0x7f1208106200 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:55.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.782+0000 7f120e000640 1 --2- 172.21.15.94:0/2382567288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1208104b90 0x7f1208104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:55.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.782+0000 7f120e000640 1 -- 172.21.15.94:0/2382567288 >> 172.21.15.94:0/2382567288 conn(0x7f1208100120 msgr2=0x7f1208102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:55.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.782+0000 7f120e000640 1 -- 172.21.15.94:0/2382567288 shutdown_connections 2024-04-26T20:07:55.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.782+0000 7f120e000640 1 -- 172.21.15.94:0/2382567288 wait complete. 2024-04-26T20:07:56.041 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.94:6838/2249987305,v1:172.21.15.94:6839/2249987305] up:boot 2024-04-26T20:07:56.071 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: fsmap 10 up:standby 2024-04-26T20:07:56.077 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T20:07:56.157 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:56.191 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:56.200 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:56.210 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:56.317 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:56.369 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:56.402 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: Deploying daemon mds.a on smithi094 2024-04-26T20:07:56.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:55 smithi103 ceph-mon[32635]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:56.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f120e000640 1 Processor -- start 2024-04-26T20:07:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f120e000640 1 -- start start 2024-04-26T20:07:56.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f120e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1208104b90 0x7f120819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:56.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f120e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1208105d80 0x7f120819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:56.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f120e000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208106740 0x7f12081a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f120e000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1208110710 con 0x7f1208106740 2024-04-26T20:07:56.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f120e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1208110590 con 0x7f1208104b90 2024-04-26T20:07:56.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f120e000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1208110890 con 0x7f1208105d80 2024-04-26T20:07:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f1207e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208106740 0x7f12081a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:56.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f1207e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208106740 0x7f12081a2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56382/0 (socket says 172.21.15.94:56382) 2024-04-26T20:07:56.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f1207e00640 1 -- 172.21.15.94:0/72521728 learned_addr learned my addr 172.21.15.94:0/72521728 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:07:56.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f1207400640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1208104b90 0x7f120819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:56.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.783+0000 7f11ffe00640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1208105d80 0x7f120819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:56.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f1207e00640 1 -- 172.21.15.94:0/72521728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1208105d80 msgr2=0x7f120819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f1207e00640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1208105d80 0x7f120819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f1207e00640 1 -- 172.21.15.94:0/72521728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1208104b90 msgr2=0x7f120819adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:56.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f1207e00640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1208104b90 0x7f120819adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:56.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f1207e00640 1 -- 172.21.15.94:0/72521728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12081a29c0 con 0x7f1208106740 2024-04-26T20:07:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f1207e00640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208106740 0x7f12081a2320 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f120000eb10 tx=0x7f120000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:56.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f1204c00640 1 -- 172.21.15.94:0/72521728 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200004b20 con 0x7f1208106740 2024-04-26T20:07:56.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f1204c00640 1 -- 172.21.15.94:0/72521728 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1200004cc0 con 0x7f1208106740 2024-04-26T20:07:56.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f1204c00640 1 -- 172.21.15.94:0/72521728 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200010670 con 0x7f1208106740 2024-04-26T20:07:56.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12081a2cb0 con 0x7f1208106740 2024-04-26T20:07:56.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.784+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12081a31b0 con 0x7f1208106740 2024-04-26T20:07:56.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.786+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f120810bf30 con 0x7f1208106740 2024-04-26T20:07:56.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.788+0000 7f1204c00640 1 -- 172.21.15.94:0/72521728 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1200005020 con 0x7f1208106740 2024-04-26T20:07:56.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.790+0000 7f1204c00640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11ec0a3bb0 0x7f11ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:07:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.790+0000 7f1204c00640 1 -- 172.21.15.94:0/72521728 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f12000d05f0 con 0x7f1208106740 2024-04-26T20:07:57.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.790+0000 7f1207400640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11ec0a3bb0 0x7f11ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:07:57.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.791+0000 7f1207400640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11ec0a3bb0 0x7f11ec0a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f120819c280 tx=0x7f11f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:07:57.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:55.794+0000 7f1204c00640 1 -- 172.21.15.94:0/72521728 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f120009d3d0 con 0x7f1208106740 2024-04-26T20:07:57.137 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.94:6838/2249987305,v1:172.21.15.94:6839/2249987305] up:boot 2024-04-26T20:07:57.162 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: fsmap 10 up:standby 2024-04-26T20:07:57.170 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T20:07:57.179 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:57.189 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:57.198 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:57.205 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:57.213 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:57.220 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:57.227 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: Deploying daemon mds.a on smithi094 2024-04-26T20:07:57.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:55 smithi094 ceph-mon[28050]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:57.246 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.94:6838/2249987305,v1:172.21.15.94:6839/2249987305] up:boot 2024-04-26T20:07:57.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: fsmap 10 up:standby 2024-04-26T20:07:57.401 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T20:07:57.409 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:57.417 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:57.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:07:57.462 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:07:57.469 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:07:57.511 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:07:57.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: Deploying daemon mds.a on smithi094 2024-04-26T20:07:57.525 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:55 smithi204 ceph-mon[32618]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:57.532 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:56 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/72521728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:57.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.046+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f120802dad0 con 0x7f1208106740 2024-04-26T20:07:57.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.047+0000 7f1204c00640 1 -- 172.21.15.94:0/72521728 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7f120009ccd0 con 0x7f1208106740 2024-04-26T20:07:57.553 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:07:57.559 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T20:07:57.567 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 11 2024-04-26T20:07:57.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.050+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11ec0a3bb0 msgr2=0x7f11ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:57.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.050+0000 7f120e000640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11ec0a3bb0 0x7f11ec0a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f120819c280 tx=0x7f11f8005f90 comp rx=0 tx=0).stop 2024-04-26T20:07:57.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.050+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208106740 msgr2=0x7f12081a2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:07:57.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.050+0000 7f120e000640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208106740 0x7f12081a2320 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f120000eb10 tx=0x7f120000c2d0 comp rx=0 tx=0).stop 2024-04-26T20:07:57.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.050+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 shutdown_connections 2024-04-26T20:07:57.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.050+0000 7f120e000640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f11ec0a3bb0 0x7f11ec0a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:57.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.050+0000 7f120e000640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1208106740 0x7f12081a2320 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:57.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.050+0000 7f120e000640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1208105d80 0x7f120819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:57.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.050+0000 7f120e000640 1 --2- 172.21.15.94:0/72521728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1208104b90 0x7f120819adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:07:57.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.051+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 >> 172.21.15.94:0/72521728 conn(0x7f1208100120 msgr2=0x7f1208109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:07:57.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.051+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 shutdown_connections 2024-04-26T20:07:57.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:07:56.051+0000 7f120e000640 1 -- 172.21.15.94:0/72521728 wait complete. 2024-04-26T20:07:57.842 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:56 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/72521728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:57.850 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:56 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/72521728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:07:58.122 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:57 smithi094 ceph-mon[28050]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:58.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:57 smithi204 ceph-mon[32618]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:57 smithi103 ceph-mon[32635]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:07:58.488 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:07:58 smithi094 systemd[1]: Starting Ceph mds.a for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:07:58.846 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:07:58 smithi094 podman[66813]: 2024-04-26 20:07:58.404251512 +0000 UTC m=+0.037399977 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:07:58.915 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:07:58 smithi094 podman[66813]: 2024-04-26 20:07:58.558013438 +0000 UTC m=+0.191161940 container create 9c1d2dbf8e40fa2d90085cf93c8b603b031b4801f3ff55d4f5176e98875bec1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T20:07:58.922 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:07:58 smithi094 podman[66813]: 2024-04-26 20:07:58.844967484 +0000 UTC m=+0.478115992 container init 9c1d2dbf8e40fa2d90085cf93c8b603b031b4801f3ff55d4f5176e98875bec1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:07:59.237 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:07:58 smithi094 podman[66813]: 2024-04-26 20:07:58.854906066 +0000 UTC m=+0.488054541 container start 9c1d2dbf8e40fa2d90085cf93c8b603b031b4801f3ff55d4f5176e98875bec1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T20:07:59.253 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:07:58 smithi094 ceph-mds[66832]: -- 172.21.15.94:0/4141001034 <== mon.0 v2:172.21.15.94:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5652858a3a40 con 0x565285877000 2024-04-26T20:07:59.261 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:07:58 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a[66827]: starting mds.a at 2024-04-26T20:07:59.270 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:07:58 smithi094 bash[66813]: 9c1d2dbf8e40fa2d90085cf93c8b603b031b4801f3ff55d4f5176e98875bec1f 2024-04-26T20:07:59.277 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:07:58 smithi094 systemd[1]: Started Ceph mds.a for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:07:59.842 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:08:00.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] up:boot 2024-04-26T20:08:00.042 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: fsmap 11 up:standby 2024-04-26T20:08:00.051 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T20:08:00.061 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:00.077 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:00.089 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:00.102 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:08:00.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:08:00.175 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:00.184 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: Deploying daemon mds.b on smithi103 2024-04-26T20:08:00.193 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:07:59 smithi103 ceph-mon[32635]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:00.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.068+0000 7f9673000640 1 -- 172.21.15.94:0/3498674282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f966c104990 msgr2=0x7f966c104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:00.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.068+0000 7f9673000640 1 --2- 172.21.15.94:0/3498674282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f966c104990 0x7f966c104db0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9660009900 tx=0x7f966002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:08:00.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.068+0000 7f9673000640 1 -- 172.21.15.94:0/3498674282 shutdown_connections 2024-04-26T20:08:00.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.068+0000 7f9673000640 1 --2- 172.21.15.94:0/3498674282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f966c106540 0x7f966c10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:00.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.068+0000 7f9673000640 1 --2- 172.21.15.94:0/3498674282 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966c105b80 0x7f966c106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:00.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.068+0000 7f9673000640 1 --2- 172.21.15.94:0/3498674282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f966c104990 0x7f966c104db0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.068+0000 7f9673000640 1 -- 172.21.15.94:0/3498674282 >> 172.21.15.94:0/3498674282 conn(0x7f966c0fff00 msgr2=0x7f966c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:00.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.069+0000 7f9673000640 1 -- 172.21.15.94:0/3498674282 shutdown_connections 2024-04-26T20:08:00.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.069+0000 7f9673000640 1 -- 172.21.15.94:0/3498674282 wait complete. 2024-04-26T20:08:00.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.070+0000 7f9673000640 1 Processor -- start 2024-04-26T20:08:00.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.070+0000 7f9673000640 1 -- start start 2024-04-26T20:08:00.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.070+0000 7f9673000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966c104990 0x7f966c19abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:00.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.070+0000 7f9673000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f966c105b80 0x7f966c19b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:00.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.070+0000 7f9673000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f966c106540 0x7f966c1a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:00.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.070+0000 7f9673000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f966c1104b0 con 0x7f966c105b80 2024-04-26T20:08:00.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.070+0000 7f9673000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f966c110330 con 0x7f966c106540 2024-04-26T20:08:00.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.070+0000 7f9673000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f966c110630 con 0x7f966c104990 2024-04-26T20:08:00.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966c104990 0x7f966c19abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:00.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f966c105b80 0x7f966c19b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:00.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966c104990 0x7f966c19abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:34974/0 (socket says 172.21.15.94:34974) 2024-04-26T20:08:00.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671c00640 1 -- 172.21.15.94:0/3853504788 learned_addr learned my addr 172.21.15.94:0/3853504788 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:00.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9672600640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f966c106540 0x7f966c1a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:00.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671c00640 1 -- 172.21.15.94:0/3853504788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f966c106540 msgr2=0x7f966c1a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:00.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671c00640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f966c106540 0x7f966c1a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:00.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671c00640 1 -- 172.21.15.94:0/3853504788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f966c105b80 msgr2=0x7f966c19b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:00.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671c00640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f966c105b80 0x7f966c19b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:00.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671c00640 1 -- 172.21.15.94:0/3853504788 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f966c1a27e0 con 0x7f966c104990 2024-04-26T20:08:00.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.071+0000 7f9671c00640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966c104990 0x7f966c19abc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9660006010 tx=0x7f96600042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:00.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.072+0000 7f966a000640 1 -- 172.21.15.94:0/3853504788 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966003f030 con 0x7f966c104990 2024-04-26T20:08:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.072+0000 7f966a000640 1 -- 172.21.15.94:0/3853504788 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9660031910 con 0x7f966c104990 2024-04-26T20:08:00.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.072+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f966c1a2a70 con 0x7f966c104990 2024-04-26T20:08:00.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.072+0000 7f966a000640 1 -- 172.21.15.94:0/3853504788 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660035070 con 0x7f966c104990 2024-04-26T20:08:00.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.072+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f966c1a2f60 con 0x7f966c104990 2024-04-26T20:08:00.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.073+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f966c10bda0 con 0x7f966c104990 2024-04-26T20:08:00.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.075+0000 7f966a000640 1 -- 172.21.15.94:0/3853504788 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9660042030 con 0x7f966c104990 2024-04-26T20:08:00.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.076+0000 7f966a000640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f96540a3bb0 0x7f96540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:00.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.076+0000 7f966a000640 1 -- 172.21.15.94:0/3853504788 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f96600f05c0 con 0x7f966c104990 2024-04-26T20:08:01.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.077+0000 7f9671200640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f96540a3bb0 0x7f96540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.077+0000 7f9671200640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f96540a3bb0 0x7f96540a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f966c19c0d0 tx=0x7f9658005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:01.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.079+0000 7f966a000640 1 -- 172.21.15.94:0/3853504788 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f96600bd3a0 con 0x7f966c104990 2024-04-26T20:08:01.056 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] up:boot 2024-04-26T20:08:01.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: fsmap 11 up:standby 2024-04-26T20:08:01.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T20:08:01.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:01.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:01.233 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:01.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:08:01.311 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:08:01.376 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:01.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: Deploying daemon mds.b on smithi103 2024-04-26T20:08:01.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:07:59 smithi204 ceph-mon[32618]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:01.446 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] up:boot 2024-04-26T20:08:01.455 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: fsmap 11 up:standby 2024-04-26T20:08:01.462 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T20:08:01.471 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:01.476 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:01.484 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:01.522 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:08:01.636 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:08:01.645 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:01.652 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: Deploying daemon mds.b on smithi103 2024-04-26T20:08:01.658 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:07:59 smithi094 ceph-mon[28050]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:01.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.322+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f966c02dc90 con 0x7f966c104990 2024-04-26T20:08:01.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.324+0000 7f966a000640 1 -- 172.21.15.94:0/3853504788 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7f96600bcca0 con 0x7f966c104990 2024-04-26T20:08:01.680 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:01.749 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-04-26T20:08:01.882 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 12 2024-04-26T20:08:01.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.326+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f96540a3bb0 msgr2=0x7f96540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:01.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.326+0000 7f9673000640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f96540a3bb0 0x7f96540a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f966c19c0d0 tx=0x7f9658005f90 comp rx=0 tx=0).stop 2024-04-26T20:08:01.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.326+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966c104990 msgr2=0x7f966c19abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:01.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.326+0000 7f9673000640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966c104990 0x7f966c19abc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9660006010 tx=0x7f96600042f0 comp rx=0 tx=0).stop 2024-04-26T20:08:01.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.327+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 shutdown_connections 2024-04-26T20:08:01.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.327+0000 7f9673000640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f96540a3bb0 0x7f96540a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.327+0000 7f9673000640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f966c106540 0x7f966c1a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:02.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.327+0000 7f9673000640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f966c105b80 0x7f966c19b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:02.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.327+0000 7f9673000640 1 --2- 172.21.15.94:0/3853504788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966c104990 0x7f966c19abc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:02.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.327+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 >> 172.21.15.94:0/3853504788 conn(0x7f966c0fff00 msgr2=0x7f966c108f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:02.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.327+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 shutdown_connections 2024-04-26T20:08:02.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:00.327+0000 7f9673000640 1 -- 172.21.15.94:0/3853504788 wait complete. 2024-04-26T20:08:02.215 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:01 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3853504788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:02.224 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:01 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3853504788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:02.233 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:01 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3853504788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:02.310 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:01 smithi103 systemd[1]: Starting Ceph mds.b for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:08:02.394 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:01 smithi103 podman[63218]: 2024-04-26 20:08:01.36370617 +0000 UTC m=+0.018045620 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:08:02.402 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:01 smithi103 podman[63218]: 2024-04-26 20:08:01.475741747 +0000 UTC m=+0.130081187 container create f37d75c0bce39d78c82cdfd5baa0c86486540f8be7f4d03d64021a855a3e56fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-26T20:08:02.411 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:01 smithi103 podman[63218]: 2024-04-26 20:08:01.703470898 +0000 UTC m=+0.357810337 container init f37d75c0bce39d78c82cdfd5baa0c86486540f8be7f4d03d64021a855a3e56fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:08:02.420 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:01 smithi103 podman[63218]: 2024-04-26 20:08:01.706114454 +0000 UTC m=+0.360453894 container start f37d75c0bce39d78c82cdfd5baa0c86486540f8be7f4d03d64021a855a3e56fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T20:08:02.429 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:01 smithi103 ceph-mds[63237]: -- 172.21.15.103:0/876626673 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556c37729a40 con 0x556c376fc800 2024-04-26T20:08:02.439 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:01 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b[63233]: starting mds.b at 2024-04-26T20:08:02.448 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:01 smithi103 bash[63218]: f37d75c0bce39d78c82cdfd5baa0c86486540f8be7f4d03d64021a855a3e56fe 2024-04-26T20:08:02.456 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:01 smithi103 systemd[1]: Started Ceph mds.b for 420d0bd6-0407-11ef-bc93-c7b262605968. 2024-04-26T20:08:02.512 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:02 smithi094 ceph-mon[28050]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:02.521 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.625 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.638 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.647 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.656 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.665 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:02 smithi204 ceph-mon[32618]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:02.674 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.734 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.785 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.819 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.826 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.835 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:02 smithi103 ceph-mon[32635]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:02.843 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.851 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.877 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:02.931 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:03.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:03 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] up:boot 2024-04-26T20:08:03.445 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:03 smithi204 ceph-mon[32618]: fsmap 12 up:standby 2024-04-26T20:08:03.457 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:08:03.467 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:08:03.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:03 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] up:boot 2024-04-26T20:08:03.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:03 smithi094 ceph-mon[28050]: fsmap 12 up:standby 2024-04-26T20:08:03.519 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:08:03.593 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:08:03.606 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:03 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] up:boot 2024-04-26T20:08:03.615 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:03 smithi103 ceph-mon[32635]: fsmap 12 up:standby 2024-04-26T20:08:03.625 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:08:03.634 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:08:04.216 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:08:04.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.417+0000 7f4297e00640 1 -- 172.21.15.94:0/3118960253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4298105ba0 msgr2=0x7f4298106020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:04.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.417+0000 7f4297e00640 1 --2- 172.21.15.94:0/3118960253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4298105ba0 0x7f4298106020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4284009900 tx=0x7f428402a410 comp rx=0 tx=0).stop 2024-04-26T20:08:04.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.417+0000 7f4297e00640 1 -- 172.21.15.94:0/3118960253 shutdown_connections 2024-04-26T20:08:04.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.417+0000 7f4297e00640 1 --2- 172.21.15.94:0/3118960253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4298106560 0x7f429810cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:04.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.417+0000 7f4297e00640 1 --2- 172.21.15.94:0/3118960253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4298105ba0 0x7f4298106020 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.417+0000 7f4297e00640 1 --2- 172.21.15.94:0/3118960253 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42981049b0 0x7f4298104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:04.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.417+0000 7f4297e00640 1 -- 172.21.15.94:0/3118960253 >> 172.21.15.94:0/3118960253 conn(0x7f42980fff40 msgr2=0x7f4298102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:04.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.417+0000 7f4297e00640 1 -- 172.21.15.94:0/3118960253 shutdown_connections 2024-04-26T20:08:04.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.417+0000 7f4297e00640 1 -- 172.21.15.94:0/3118960253 wait complete. 2024-04-26T20:08:04.644 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:04 smithi204 ceph-mon[32618]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:04.654 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:04.744 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:04.754 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:04.763 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:04.770 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:04 smithi103 ceph-mon[32635]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:04.778 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:04.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:04.792 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:04.856 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:04.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.418+0000 7f4297e00640 1 Processor -- start 2024-04-26T20:08:04.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.418+0000 7f4297e00640 1 -- start start 2024-04-26T20:08:04.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.418+0000 7f4297e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42981049b0 0x7f429819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:04.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.418+0000 7f4297e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4298105ba0 0x7f429819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:05.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.418+0000 7f4297e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4298106560 0x7f42981a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:05.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.418+0000 7f4297e00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4298110510 con 0x7f4298105ba0 2024-04-26T20:08:05.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.418+0000 7f4297e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4298110390 con 0x7f42981049b0 2024-04-26T20:08:05.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.418+0000 7f4297e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4298110690 con 0x7f4298106560 2024-04-26T20:08:05.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4297400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4298106560 0x7f42981a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:05.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4297400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4298106560 0x7f42981a2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:41894/0 (socket says 172.21.15.94:41894) 2024-04-26T20:08:05.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4297400640 1 -- 172.21.15.94:0/1748703429 learned_addr learned my addr 172.21.15.94:0/1748703429 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:05.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4296000640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4298105ba0 0x7f429819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:05.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4296a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42981049b0 0x7f429819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4297400640 1 -- 172.21.15.94:0/1748703429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42981049b0 msgr2=0x7f429819ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4297400640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42981049b0 0x7f429819ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:05.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4297400640 1 -- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4298105ba0 msgr2=0x7f429819b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:05.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4297400640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4298105ba0 0x7f429819b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:05.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4297400640 1 -- 172.21.15.94:0/1748703429 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42981a2780 con 0x7f4298106560 2024-04-26T20:08:05.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.419+0000 7f4296a00640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42981049b0 0x7f429819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:05.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.420+0000 7f4297400640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4298106560 0x7f42981a2110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f429000eb10 tx=0x7f429000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:05.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.420+0000 7f428f400640 1 -- 172.21.15.94:0/1748703429 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290004b20 con 0x7f4298106560 2024-04-26T20:08:05.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.420+0000 7f428f400640 1 -- 172.21.15.94:0/1748703429 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4290004cc0 con 0x7f4298106560 2024-04-26T20:08:05.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.420+0000 7f428f400640 1 -- 172.21.15.94:0/1748703429 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290010670 con 0x7f4298106560 2024-04-26T20:08:05.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.420+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4298076dd0 con 0x7f4298106560 2024-04-26T20:08:05.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.420+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42980772a0 con 0x7f4298106560 2024-04-26T20:08:05.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.421+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f429810bcd0 con 0x7f4298106560 2024-04-26T20:08:05.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.423+0000 7f428f400640 1 -- 172.21.15.94:0/1748703429 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4290005020 con 0x7f4298106560 2024-04-26T20:08:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.424+0000 7f428f400640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f427c0a3bb0 0x7f427c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:05.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.424+0000 7f428f400640 1 -- 172.21.15.94:0/1748703429 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f42900d05b0 con 0x7f4298106560 2024-04-26T20:08:05.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.425+0000 7f4296a00640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f427c0a3bb0 0x7f427c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:05.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.425+0000 7f4296a00640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f427c0a3bb0 0x7f427c0a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4298105a00 tx=0x7f4288005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:05.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.428+0000 7f428f400640 1 -- 172.21.15.94:0/1748703429 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f429009d390 con 0x7f4298106560 2024-04-26T20:08:05.702 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:04 smithi094 ceph-mon[28050]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:05.748 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:05.755 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:05.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:05.769 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:05.776 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:05 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1748703429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:05.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:05.914 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:05.921 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:05.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:08:05.945 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:06.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:05 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1748703429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:06.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:06.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:06.096 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:06.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:08:06.115 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:06.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.719+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4298063530 con 0x7f4298106560 2024-04-26T20:08:06.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.721+0000 7f428f400640 1 -- 172.21.15.94:0/1748703429 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10488 (secure 0 0 0) 0x7f429009cc90 con 0x7f4298106560 2024-04-26T20:08:06.181 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:06.190 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-26T20:08:06.358 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 13 2024-04-26T20:08:06.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f427c0a3bb0 msgr2=0x7f427c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:06.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f427c0a3bb0 0x7f427c0a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4298105a00 tx=0x7f4288005f90 comp rx=0 tx=0).stop 2024-04-26T20:08:06.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4298106560 msgr2=0x7f42981a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:06.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4298106560 0x7f42981a2110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f429000eb10 tx=0x7f429000c2d0 comp rx=0 tx=0).stop 2024-04-26T20:08:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 shutdown_connections 2024-04-26T20:08:06.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f427c0a3bb0 0x7f427c0a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:06.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4298106560 0x7f42981a2110 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:06.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4298105ba0 0x7f429819b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:06.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 --2- 172.21.15.94:0/1748703429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42981049b0 0x7f429819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:06.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.723+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 >> 172.21.15.94:0/1748703429 conn(0x7f42980fff40 msgr2=0x7f4298108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:06.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.724+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 shutdown_connections 2024-04-26T20:08:06.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:04.724+0000 7f4297e00640 1 -- 172.21.15.94:0/1748703429 wait complete. 2024-04-26T20:08:06.548 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:08:06.607 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:05 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1748703429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:06.615 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:06.621 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:06.722 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:06.752 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:08:06.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:06.769 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:08:06.777 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:06 smithi094 ceph-mon[28050]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:06.785 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:08:06.794 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:06 smithi204 ceph-mon[32618]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:06.852 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:08:06.880 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:06 smithi103 ceph-mon[32635]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:07.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.096+0000 7fda7a200640 1 -- 172.21.15.94:0/3916881391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda7410ab40 msgr2=0x7fda741113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:07.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.096+0000 7fda7a200640 1 --2- 172.21.15.94:0/3916881391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda7410ab40 0x7fda741113d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009900 tx=0x7fda6c02a410 comp rx=0 tx=0).stop 2024-04-26T20:08:07.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.096+0000 7fda7a200640 1 -- 172.21.15.94:0/3916881391 shutdown_connections 2024-04-26T20:08:07.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.096+0000 7fda7a200640 1 --2- 172.21.15.94:0/3916881391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda7410ab40 0x7fda741113d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:07.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.096+0000 7fda7a200640 1 --2- 172.21.15.94:0/3916881391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda7410a180 0x7fda7410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:07.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.096+0000 7fda7a200640 1 --2- 172.21.15.94:0/3916881391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda74108f90 0x7fda741093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:07.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.096+0000 7fda7a200640 1 -- 172.21.15.94:0/3916881391 >> 172.21.15.94:0/3916881391 conn(0x7fda74077f40 msgr2=0x7fda7407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:07.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.096+0000 7fda7a200640 1 -- 172.21.15.94:0/3916881391 shutdown_connections 2024-04-26T20:08:07.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.096+0000 7fda7a200640 1 -- 172.21.15.94:0/3916881391 wait complete. 2024-04-26T20:08:07.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.097+0000 7fda7a200640 1 Processor -- start 2024-04-26T20:08:07.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda7a200640 1 -- start start 2024-04-26T20:08:07.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda7a200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda74108f90 0x7fda7419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:07.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda7a200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7410a180 0x7fda7419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:07.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda7a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda7410ab40 0x7fda741a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:07.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda7a200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fda74114b20 con 0x7fda7410a180 2024-04-26T20:08:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda7a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda741149a0 con 0x7fda7410ab40 2024-04-26T20:08:07.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda7a200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fda74114ca0 con 0x7fda74108f90 2024-04-26T20:08:07.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda72a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7410a180 0x7fda7419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:07.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda73e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda7410ab40 0x7fda741a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:07.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda73e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda7410ab40 0x7fda741a66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:44130/0 (socket says 172.21.15.94:44130) 2024-04-26T20:08:07.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda73e00640 1 -- 172.21.15.94:0/2388442251 learned_addr learned my addr 172.21.15.94:0/2388442251 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:07.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda73400640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda74108f90 0x7fda7419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:07.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda73e00640 1 -- 172.21.15.94:0/2388442251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda74108f90 msgr2=0x7fda7419f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:07.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda73e00640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda74108f90 0x7fda7419f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:07.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda73e00640 1 -- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7410a180 msgr2=0x7fda7419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:07.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda73e00640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7410a180 0x7fda7419f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:07.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda73e00640 1 -- 172.21.15.94:0/2388442251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda741a6d60 con 0x7fda7410ab40 2024-04-26T20:08:07.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.098+0000 7fda72a00640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7410a180 0x7fda7419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:07.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.099+0000 7fda73e00640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda7410ab40 0x7fda741a66f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fda6c005ab0 tx=0x7fda6c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:07.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.099+0000 7fda6be00640 1 -- 172.21.15.94:0/2388442251 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c03f070 con 0x7fda7410ab40 2024-04-26T20:08:07.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.100+0000 7fda6be00640 1 -- 172.21.15.94:0/2388442251 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda6c031c50 con 0x7fda7410ab40 2024-04-26T20:08:07.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.100+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda741a6ff0 con 0x7fda7410ab40 2024-04-26T20:08:07.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.100+0000 7fda6be00640 1 -- 172.21.15.94:0/2388442251 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c035030 con 0x7fda7410ab40 2024-04-26T20:08:07.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.100+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda741a74b0 con 0x7fda7410ab40 2024-04-26T20:08:07.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.101+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda74110250 con 0x7fda7410ab40 2024-04-26T20:08:07.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.103+0000 7fda6be00640 1 -- 172.21.15.94:0/2388442251 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fda6c042070 con 0x7fda7410ab40 2024-04-26T20:08:07.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.104+0000 7fda6be00640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fda580a3c30 0x7fda580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:08.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.104+0000 7fda6be00640 1 -- 172.21.15.94:0/2388442251 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fda6c0f0660 con 0x7fda7410ab40 2024-04-26T20:08:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.105+0000 7fda73400640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fda580a3c30 0x7fda580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:08.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.105+0000 7fda73400640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fda580a3c30 0x7fda580a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fda74109fe0 tx=0x7fda64009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:08.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.109+0000 7fda6be00640 1 -- 172.21.15.94:0/2388442251 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fda6c0bcdf0 con 0x7fda7410ab40 2024-04-26T20:08:08.206 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:07 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.214 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:07 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.227 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:07 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.236 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:07 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.244 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:07 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.251 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:07 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2388442251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:08.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:07 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.270 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:07 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:07 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:07 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:07 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.427 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:07 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2388442251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:08.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.324+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda74063530 con 0x7fda7410ab40 2024-04-26T20:08:08.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.326+0000 7fda6be00640 1 -- 172.21.15.94:0/2388442251 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18493 (secure 0 0 0) 0x7fda6c0c10f0 con 0x7fda7410ab40 2024-04-26T20:08:08.593 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:08.652 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":81,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:07:11.294222+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:08:08.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fda580a3c30 msgr2=0x7fda580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:08.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fda580a3c30 0x7fda580a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fda74109fe0 tx=0x7fda64009500 comp rx=0 tx=0).stop 2024-04-26T20:08:08.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda7410ab40 msgr2=0x7fda741a66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:08.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda7410ab40 0x7fda741a66f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fda6c005ab0 tx=0x7fda6c003b90 comp rx=0 tx=0).stop 2024-04-26T20:08:08.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 shutdown_connections 2024-04-26T20:08:08.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fda580a3c30 0x7fda580a60f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:08.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda7410ab40 0x7fda741a66f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:08.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda7410a180 0x7fda7419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:08.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 --2- 172.21.15.94:0/2388442251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda74108f90 0x7fda7419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 >> 172.21.15.94:0/2388442251 conn(0x7fda74077f40 msgr2=0x7fda7410d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:08.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 shutdown_connections 2024-04-26T20:08:08.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:07.328+0000 7fda7a200640 1 -- 172.21.15.94:0/2388442251 wait complete. 2024-04-26T20:08:08.835 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T20:08:08.854 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:08:08.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:07 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.974 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:07 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:08.983 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:07 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.062 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:07 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.071 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:07 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.082 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:07 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2388442251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:09.092 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:08 smithi103 ceph-mon[32635]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:09.100 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.107 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.115 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:09.123 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:08:09.137 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.239 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:08 smithi204 ceph-mon[32618]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:09.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.255 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:09.324 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:08:09.472 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.486 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:08 smithi094 ceph-mon[28050]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:09.494 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:08:09.516 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:08:09.521 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:08:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.308+0000 7f8c98600640 1 -- 172.21.15.94:0/1287755431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c9010a180 msgr2=0x7f8c9010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:09.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.308+0000 7f8c98600640 1 --2- 172.21.15.94:0/1287755431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c9010a180 0x7f8c9010a600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009860 tx=0x7f8c7c02a340 comp rx=0 tx=0).stop 2024-04-26T20:08:09.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.309+0000 7f8c98600640 1 -- 172.21.15.94:0/1287755431 shutdown_connections 2024-04-26T20:08:09.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.309+0000 7f8c98600640 1 --2- 172.21.15.94:0/1287755431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c9010ab40 0x7f8c901113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:09.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.309+0000 7f8c98600640 1 --2- 172.21.15.94:0/1287755431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c9010a180 0x7f8c9010a600 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:09.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.309+0000 7f8c98600640 1 --2- 172.21.15.94:0/1287755431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90108f90 0x7f8c901093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:09.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.309+0000 7f8c98600640 1 -- 172.21.15.94:0/1287755431 >> 172.21.15.94:0/1287755431 conn(0x7f8c90077f30 msgr2=0x7f8c9007a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:09.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.309+0000 7f8c98600640 1 -- 172.21.15.94:0/1287755431 shutdown_connections 2024-04-26T20:08:09.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.309+0000 7f8c98600640 1 -- 172.21.15.94:0/1287755431 wait complete. 2024-04-26T20:08:09.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c98600640 1 Processor -- start 2024-04-26T20:08:09.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c98600640 1 -- start start 2024-04-26T20:08:10.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c98600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90108f90 0x7f8c9019f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:10.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c98600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c9010a180 0x7f8c9019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:10.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c98600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c9010ab40 0x7f8c901a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:10.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c98600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8c90114ad0 con 0x7f8c9010a180 2024-04-26T20:08:10.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c98600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8c90114950 con 0x7f8c90108f90 2024-04-26T20:08:10.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c98600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8c90114c50 con 0x7f8c9010ab40 2024-04-26T20:08:10.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c95200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c9010a180 0x7f8c9019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c95200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c9010a180 0x7f8c9019f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60166/0 (socket says 172.21.15.94:60166) 2024-04-26T20:08:10.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.310+0000 7f8c95200640 1 -- 172.21.15.94:0/1751330791 learned_addr learned my addr 172.21.15.94:0/1751330791 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:10.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c96600640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c9010ab40 0x7f8c901a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:10.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c95c00640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90108f90 0x7f8c9019f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:10.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c95200640 1 -- 172.21.15.94:0/1751330791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c9010ab40 msgr2=0x7f8c901a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:10.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c95200640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c9010ab40 0x7f8c901a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:10.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c95200640 1 -- 172.21.15.94:0/1751330791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90108f90 msgr2=0x7f8c9019f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c95200640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90108f90 0x7f8c9019f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:10.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c95200640 1 -- 172.21.15.94:0/1751330791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c901a6de0 con 0x7f8c9010a180 2024-04-26T20:08:10.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c95c00640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90108f90 0x7f8c9019f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c96600640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c9010ab40 0x7f8c901a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:10.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.311+0000 7f8c95200640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c9010a180 0x7f8c9019f700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0057c0 tx=0x7f8c7c032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.312+0000 7f8c8e000640 1 -- 172.21.15.94:0/1751330791 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c037040 con 0x7f8c9010a180 2024-04-26T20:08:10.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.312+0000 7f8c8e000640 1 -- 172.21.15.94:0/1751330791 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c7c030cd0 con 0x7f8c9010a180 2024-04-26T20:08:10.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.312+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c901a7070 con 0x7f8c9010a180 2024-04-26T20:08:10.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.312+0000 7f8c8e000640 1 -- 172.21.15.94:0/1751330791 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c005350 con 0x7f8c9010a180 2024-04-26T20:08:10.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.312+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c901a7500 con 0x7f8c9010a180 2024-04-26T20:08:10.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.313+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c90110320 con 0x7f8c9010a180 2024-04-26T20:08:10.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.315+0000 7f8c8e000640 1 -- 172.21.15.94:0/1751330791 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8c7c043020 con 0x7f8c9010a180 2024-04-26T20:08:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.316+0000 7f8c8e000640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8c780a3bb0 0x7f8c780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.316+0000 7f8c8e000640 1 -- 172.21.15.94:0/1751330791 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8c7c0ef6e0 con 0x7f8c9010a180 2024-04-26T20:08:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.316+0000 7f8c95c00640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8c780a3bb0 0x7f8c780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.317+0000 7f8c95c00640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8c780a3bb0 0x7f8c780a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8c90109fe0 tx=0x7f8c8400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:10.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.321+0000 7f8c8e000640 1 -- 172.21.15.94:0/1751330791 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8c7c0bc4c0 con 0x7f8c9010a180 2024-04-26T20:08:10.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.523+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8c90063530 con 0x7f8c9010a180 2024-04-26T20:08:10.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.524+0000 7f8c8e000640 1 -- 172.21.15.94:0/1751330791 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f8c7c0bbdc0 con 0x7f8c9010a180 2024-04-26T20:08:10.917 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:08:11.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8c780a3bb0 msgr2=0x7f8c780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:11.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8c780a3bb0 0x7f8c780a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8c90109fe0 tx=0x7f8c8400b390 comp rx=0 tx=0).stop 2024-04-26T20:08:11.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c9010a180 msgr2=0x7f8c9019f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c9010a180 0x7f8c9019f700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0057c0 tx=0x7f8c7c032eb0 comp rx=0 tx=0).stop 2024-04-26T20:08:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 shutdown_connections 2024-04-26T20:08:11.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8c780a3bb0 0x7f8c780a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:11.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c9010ab40 0x7f8c901a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:11.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c9010a180 0x7f8c9019f700 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:11.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 --2- 172.21.15.94:0/1751330791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c90108f90 0x7f8c9019f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:11.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.527+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 >> 172.21.15.94:0/1751330791 conn(0x7f8c90077f30 msgr2=0x7f8c9010d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:11.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.528+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 shutdown_connections 2024-04-26T20:08:11.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:09.528+0000 7f8c98600640 1 -- 172.21.15.94:0/1751330791 wait complete. 2024-04-26T20:08:11.352 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T20:08:11.359 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:10 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1751330791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:08:11.426 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:10 smithi094 ceph-mon[28050]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:11.436 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:10 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1751330791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:08:11.444 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:10 smithi204 ceph-mon[32618]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:11.516 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:10 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1751330791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:08:11.524 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:10 smithi103 ceph-mon[32635]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:11.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.650+0000 7f09b3a00640 1 -- 172.21.15.94:0/2186979254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac078530 msgr2=0x7f09ac078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:11.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.650+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2186979254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac078530 0x7f09ac078990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0998009900 tx=0x7f099802a410 comp rx=0 tx=0).stop 2024-04-26T20:08:11.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.650+0000 7f09b3a00640 1 -- 172.21.15.94:0/2186979254 shutdown_connections 2024-04-26T20:08:11.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.650+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2186979254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09ac078ed0 0x7f09ac079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:11.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.650+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2186979254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac078530 0x7f09ac078990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:11.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.650+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2186979254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09ac0772f0 0x7f09ac077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:11.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.650+0000 7f09b3a00640 1 -- 172.21.15.94:0/2186979254 >> 172.21.15.94:0/2186979254 conn(0x7f09ac100120 msgr2=0x7f09ac102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.650+0000 7f09b3a00640 1 -- 172.21.15.94:0/2186979254 shutdown_connections 2024-04-26T20:08:11.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.650+0000 7f09b3a00640 1 -- 172.21.15.94:0/2186979254 wait complete. 2024-04-26T20:08:11.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.651+0000 7f09b3a00640 1 Processor -- start 2024-04-26T20:08:11.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.651+0000 7f09b3a00640 1 -- start start 2024-04-26T20:08:11.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b3a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac0772f0 0x7f09ac19f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:11.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b3a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09ac078530 0x7f09ac19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:12.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b3a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09ac078ed0 0x7f09ac1a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:12.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b3a00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac114af0 con 0x7f09ac0772f0 2024-04-26T20:08:12.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b3a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac114970 con 0x7f09ac078ed0 2024-04-26T20:08:12.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b3a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac114c70 con 0x7f09ac078530 2024-04-26T20:08:12.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b1000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac0772f0 0x7f09ac19f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:12.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b1000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac0772f0 0x7f09ac19f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59634/0 (socket says 172.21.15.94:59634) 2024-04-26T20:08:12.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b1000640 1 -- 172.21.15.94:0/2832297885 learned_addr learned my addr 172.21.15.94:0/2832297885 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:12.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09b1a00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09ac078ed0 0x7f09ac1a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:12.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.652+0000 7f09abe00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09ac078530 0x7f09ac19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:12.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09b1000640 1 -- 172.21.15.94:0/2832297885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09ac078530 msgr2=0x7f09ac19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:12.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09b1000640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09ac078530 0x7f09ac19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:12.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09b1000640 1 -- 172.21.15.94:0/2832297885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09ac078ed0 msgr2=0x7f09ac1a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:12.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09b1000640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09ac078ed0 0x7f09ac1a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:12.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09b1000640 1 -- 172.21.15.94:0/2832297885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac1a6de0 con 0x7f09ac0772f0 2024-04-26T20:08:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09abe00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09ac078530 0x7f09ac19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:08:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09b1000640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac0772f0 0x7f09ac19f1c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e9f0 tx=0x7f09a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:12.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09a9600640 1 -- 172.21.15.94:0/2832297885 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a000cd80 con 0x7f09ac0772f0 2024-04-26T20:08:12.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09a9600640 1 -- 172.21.15.94:0/2832297885 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09a00108e0 con 0x7f09ac0772f0 2024-04-26T20:08:12.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09ac1a70d0 con 0x7f09ac0772f0 2024-04-26T20:08:12.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09a9600640 1 -- 172.21.15.94:0/2832297885 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0018980 con 0x7f09ac0772f0 2024-04-26T20:08:12.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.653+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09ac1a7640 con 0x7f09ac0772f0 2024-04-26T20:08:12.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.654+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09ac111720 con 0x7f09ac0772f0 2024-04-26T20:08:12.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.656+0000 7f09a9600640 1 -- 172.21.15.94:0/2832297885 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f09a0010430 con 0x7f09ac0772f0 2024-04-26T20:08:12.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.658+0000 7f09a9600640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f09940a3c30 0x7f09940a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:12.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.658+0000 7f09a9600640 1 -- 172.21.15.94:0/2832297885 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f09a00d0370 con 0x7f09ac0772f0 2024-04-26T20:08:12.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.658+0000 7f09abe00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f09940a3c30 0x7f09940a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:12.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.659+0000 7f09abe00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f09940a3c30 0x7f09940a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0998000c00 tx=0x7f09980314c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:12.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.662+0000 7f09a9600640 1 -- 172.21.15.94:0/2832297885 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f09a009d150 con 0x7f09ac0772f0 2024-04-26T20:08:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.864+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f09ac063530 con 0x7f09ac0772f0 2024-04-26T20:08:12.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.865+0000 7f09a9600640 1 -- 172.21.15.94:0/2832297885 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f09a009ca50 con 0x7f09ac0772f0 2024-04-26T20:08:12.706 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:12.725 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-04-26T20:08:12.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-04-26T20:08:12.775 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-04-26T20:08:12.782 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-04-26T20:08:12.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-26T20:08:12.876 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-04-26T20:08:12.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.867+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f09940a3c30 msgr2=0x7f09940a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:12.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.867+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f09940a3c30 0x7f09940a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0998000c00 tx=0x7f09980314c0 comp rx=0 tx=0).stop 2024-04-26T20:08:12.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.867+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac0772f0 msgr2=0x7f09ac19f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:12.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.867+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac0772f0 0x7f09ac19f1c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e9f0 tx=0x7f09a000eeb0 comp rx=0 tx=0).stop 2024-04-26T20:08:12.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.867+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 shutdown_connections 2024-04-26T20:08:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.867+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f09940a3c30 0x7f09940a60f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:12.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.867+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09ac078ed0 0x7f09ac1a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:12.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.868+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09ac078530 0x7f09ac19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:12.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.868+0000 7f09b3a00640 1 --2- 172.21.15.94:0/2832297885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09ac0772f0 0x7f09ac19f1c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:13.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.868+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 >> 172.21.15.94:0/2832297885 conn(0x7f09ac100120 msgr2=0x7f09ac1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:13.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.868+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 shutdown_connections 2024-04-26T20:08:13.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:11.868+0000 7f09b3a00640 1 -- 172.21.15.94:0/2832297885 wait complete. 2024-04-26T20:08:13.205 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T20:08:13.250 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T20:08:13.263 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:12 smithi103 ceph-mon[32635]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:13.364 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:12 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2832297885' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T20:08:13.370 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:12 smithi204 ceph-mon[32618]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:13.378 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:12 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2832297885' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T20:08:13.386 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:12 smithi094 ceph-mon[28050]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:13.394 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:12 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2832297885' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T20:08:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.666+0000 7fd455c00640 1 -- 172.21.15.94:0/2355147097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd450109fc0 msgr2=0x7fd45010a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:13.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.666+0000 7fd455c00640 1 --2- 172.21.15.94:0/2355147097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd450109fc0 0x7fd45010a440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd43c009ca0 tx=0x7fd43c02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:08:13.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.666+0000 7fd455c00640 1 -- 172.21.15.94:0/2355147097 shutdown_connections 2024-04-26T20:08:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.666+0000 7fd455c00640 1 --2- 172.21.15.94:0/2355147097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd45010a980 0x7fd4501112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:13.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.666+0000 7fd455c00640 1 --2- 172.21.15.94:0/2355147097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd450109fc0 0x7fd45010a440 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:13.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.666+0000 7fd455c00640 1 --2- 172.21.15.94:0/2355147097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd450108e70 0x7fd450109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:13.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.666+0000 7fd455c00640 1 -- 172.21.15.94:0/2355147097 >> 172.21.15.94:0/2355147097 conn(0x7fd450077e00 msgr2=0x7fd45007a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:13.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.666+0000 7fd455c00640 1 -- 172.21.15.94:0/2355147097 shutdown_connections 2024-04-26T20:08:13.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.666+0000 7fd455c00640 1 -- 172.21.15.94:0/2355147097 wait complete. 2024-04-26T20:08:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.667+0000 7fd455c00640 1 Processor -- start 2024-04-26T20:08:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.667+0000 7fd455c00640 1 -- start start 2024-04-26T20:08:13.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd450108e70 0x7fd45019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:13.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd450109fc0 0x7fd45019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:13.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455c00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd45010a980 0x7fd4501a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:14.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455c00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd450114a20 con 0x7fd45010a980 2024-04-26T20:08:14.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501148a0 con 0x7fd450108e70 2024-04-26T20:08:14.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd450114ba0 con 0x7fd450109fc0 2024-04-26T20:08:14.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd45010a980 0x7fd4501a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:14.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd45010a980 0x7fd4501a6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59656/0 (socket says 172.21.15.94:59656) 2024-04-26T20:08:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455200640 1 -- 172.21.15.94:0/4229852213 learned_addr learned my addr 172.21.15.94:0/4229852213 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:14.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd44fe00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd450108e70 0x7fd45019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:14.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd44f400640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd450109fc0 0x7fd45019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:14.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd44fe00640 1 -- 172.21.15.94:0/4229852213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd450109fc0 msgr2=0x7fd45019f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:14.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd44fe00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd450109fc0 0x7fd45019f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:14.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd44fe00640 1 -- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd45010a980 msgr2=0x7fd4501a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:14.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd44fe00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd45010a980 0x7fd4501a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:14.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd44fe00640 1 -- 172.21.15.94:0/4229852213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4501a6c80 con 0x7fd450108e70 2024-04-26T20:08:14.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd455200640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd45010a980 0x7fd4501a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:14.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.668+0000 7fd44f400640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd450109fc0 0x7fd45019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:14.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.669+0000 7fd44fe00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd450108e70 0x7fd45019f060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd44000e9f0 tx=0x7fd44000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:14.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.669+0000 7fd44cc00640 1 -- 172.21.15.94:0/4229852213 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44000ccc0 con 0x7fd450108e70 2024-04-26T20:08:14.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.669+0000 7fd44cc00640 1 -- 172.21.15.94:0/4229852213 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4400108e0 con 0x7fd450108e70 2024-04-26T20:08:14.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.669+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4501a6f70 con 0x7fd450108e70 2024-04-26T20:08:14.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.669+0000 7fd44cc00640 1 -- 172.21.15.94:0/4229852213 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440018880 con 0x7fd450108e70 2024-04-26T20:08:14.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.669+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4501a73d0 con 0x7fd450108e70 2024-04-26T20:08:14.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.670+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd450110180 con 0x7fd450108e70 2024-04-26T20:08:14.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.672+0000 7fd44cc00640 1 -- 172.21.15.94:0/4229852213 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd440010430 con 0x7fd450108e70 2024-04-26T20:08:14.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.673+0000 7fd44cc00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd4340a3bb0 0x7fd4340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:14.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.674+0000 7fd44cc00640 1 -- 172.21.15.94:0/4229852213 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd4400d02f0 con 0x7fd450108e70 2024-04-26T20:08:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.674+0000 7fd44f400640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd4340a3bb0 0x7fd4340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:14.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.674+0000 7fd44f400640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd4340a3bb0 0x7fd4340a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd43c000f80 tx=0x7fd43c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:14.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.678+0000 7fd44cc00640 1 -- 172.21.15.94:0/4229852213 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd44009ca80 con 0x7fd450108e70 2024-04-26T20:08:14.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:13.885+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd450063560 con 0x7fd450108e70 2024-04-26T20:08:14.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.614+0000 7fd44cc00640 1 -- 172.21.15.94:0/4229852213 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fd4400a0d80 con 0x7fd450108e70 2024-04-26T20:08:14.803 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-04-26T20:08:14.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd4340a3bb0 msgr2=0x7fd4340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:14.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd4340a3bb0 0x7fd4340a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd43c000f80 tx=0x7fd43c004a40 comp rx=0 tx=0).stop 2024-04-26T20:08:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd450108e70 msgr2=0x7fd45019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:14.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd450108e70 0x7fd45019f060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd44000e9f0 tx=0x7fd44000eeb0 comp rx=0 tx=0).stop 2024-04-26T20:08:14.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 shutdown_connections 2024-04-26T20:08:14.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd4340a3bb0 0x7fd4340a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:14.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd45010a980 0x7fd4501a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:14.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd450109fc0 0x7fd45019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:14.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 --2- 172.21.15.94:0/4229852213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd450108e70 0x7fd45019f060 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:14.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 >> 172.21.15.94:0/4229852213 conn(0x7fd450077e00 msgr2=0x7fd45010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:14.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 shutdown_connections 2024-04-26T20:08:14.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:14.620+0000 7fd455c00640 1 -- 172.21.15.94:0/4229852213 wait complete. 2024-04-26T20:08:15.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-26T20:08:15.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:14 smithi094 ceph-mon[28050]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:15.105 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:14 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4229852213' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:08:15.166 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:14 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:08:15.217 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:14 smithi204 ceph-mon[32618]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:15.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:14 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4229852213' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:08:15.369 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:14 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:08:15.422 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:14 smithi103 ceph-mon[32635]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:15.430 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:14 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4229852213' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:08:15.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:14 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:08:15.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.428+0000 7f3415000640 1 -- 172.21.15.94:0/569104532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3410076970 msgr2=0x7f3410074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:15.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.428+0000 7f3415000640 1 --2- 172.21.15.94:0/569104532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3410076970 0x7f3410074dc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3400009900 tx=0x7f340002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:08:15.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.428+0000 7f3415000640 1 -- 172.21.15.94:0/569104532 shutdown_connections 2024-04-26T20:08:15.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.428+0000 7f3415000640 1 --2- 172.21.15.94:0/569104532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f341010ee90 0x7f3410111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:15.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.428+0000 7f3415000640 1 --2- 172.21.15.94:0/569104532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3410075390 0x7f3410075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:15.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.428+0000 7f3415000640 1 --2- 172.21.15.94:0/569104532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3410076970 0x7f3410074dc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:15.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.428+0000 7f3415000640 1 -- 172.21.15.94:0/569104532 >> 172.21.15.94:0/569104532 conn(0x7f34100fde30 msgr2=0x7f3410100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:15.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.428+0000 7f3415000640 1 -- 172.21.15.94:0/569104532 shutdown_connections 2024-04-26T20:08:15.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.428+0000 7f3415000640 1 -- 172.21.15.94:0/569104532 wait complete. 2024-04-26T20:08:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.429+0000 7f3415000640 1 Processor -- start 2024-04-26T20:08:15.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.429+0000 7f3415000640 1 -- start start 2024-04-26T20:08:15.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3415000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3410075390 0x7f341019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:15.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3415000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3410076970 0x7f341019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:15.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3415000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f341010ee90 0x7f34101a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:15.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3415000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f34101149c0 con 0x7f3410076970 2024-04-26T20:08:16.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3415000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3410114840 con 0x7f3410075390 2024-04-26T20:08:16.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3415000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3410114b40 con 0x7f341010ee90 2024-04-26T20:08:16.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3407e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3410076970 0x7f341019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:16.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3407e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3410076970 0x7f341019f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59682/0 (socket says 172.21.15.94:59682) 2024-04-26T20:08:16.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3407e00640 1 -- 172.21.15.94:0/1521407073 learned_addr learned my addr 172.21.15.94:0/1521407073 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:16.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f340fe00640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f341010ee90 0x7f34101a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:16.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f340f400640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3410075390 0x7f341019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:16.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3407e00640 1 -- 172.21.15.94:0/1521407073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f341010ee90 msgr2=0x7f34101a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:16.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3407e00640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f341010ee90 0x7f34101a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3407e00640 1 -- 172.21.15.94:0/1521407073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3410075390 msgr2=0x7f341019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:16.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3407e00640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3410075390 0x7f341019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:16.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f3407e00640 1 -- 172.21.15.94:0/1521407073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34101a6ce0 con 0x7f3410076970 2024-04-26T20:08:16.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.430+0000 7f340fe00640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f341010ee90 0x7f34101a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:16.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.431+0000 7f3407e00640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3410076970 0x7f341019f600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f33f800e9f0 tx=0x7f33f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:16.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.431+0000 7f340cc00640 1 -- 172.21.15.94:0/1521407073 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f800cd80 con 0x7f3410076970 2024-04-26T20:08:16.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.431+0000 7f340cc00640 1 -- 172.21.15.94:0/1521407073 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33f80054d0 con 0x7f3410076970 2024-04-26T20:08:16.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.431+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34101a6fd0 con 0x7f3410076970 2024-04-26T20:08:16.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.431+0000 7f340cc00640 1 -- 172.21.15.94:0/1521407073 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8010690 con 0x7f3410076970 2024-04-26T20:08:16.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.431+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34101a7490 con 0x7f3410076970 2024-04-26T20:08:16.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.432+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3410105060 con 0x7f3410076970 2024-04-26T20:08:16.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.435+0000 7f340cc00640 1 -- 172.21.15.94:0/1521407073 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f33f8004e30 con 0x7f3410076970 2024-04-26T20:08:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.437+0000 7f340cc00640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33f40a3c30 0x7f33f40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.437+0000 7f340cc00640 1 -- 172.21.15.94:0/1521407073 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f33f80d06b0 con 0x7f3410076970 2024-04-26T20:08:16.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.438+0000 7f340f400640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33f40a3c30 0x7f33f40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:16.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.438+0000 7f340f400640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33f40a3c30 0x7f33f40a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3400005ab0 tx=0x7f3400005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:16.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.439+0000 7f340cc00640 1 -- 172.21.15.94:0/1521407073 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f33f8054060 con 0x7f3410076970 2024-04-26T20:08:16.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:15 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T20:08:16.653 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:15 smithi204 ceph-mon[32618]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T20:08:16.664 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:15 smithi204 ceph-mon[32618]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T20:08:16.736 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:15 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T20:08:16.768 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:15 smithi103 ceph-mon[32635]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T20:08:16.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:15 smithi103 ceph-mon[32635]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T20:08:16.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:15.659+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f341002dad0 con 0x7f3410076970 2024-04-26T20:08:16.967 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:15 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T20:08:16.973 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:15 smithi094 ceph-mon[28050]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T20:08:16.980 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:15 smithi094 ceph-mon[28050]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T20:08:16.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.616+0000 7f340cc00640 1 -- 172.21.15.94:0/1521407073 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7f33f809d320 con 0x7f3410076970 2024-04-26T20:08:16.993 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-04-26T20:08:16.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.619+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33f40a3c30 msgr2=0x7f33f40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.619+0000 7f3415000640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33f40a3c30 0x7f33f40a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3400005ab0 tx=0x7f3400005a20 comp rx=0 tx=0).stop 2024-04-26T20:08:17.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.619+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3410076970 msgr2=0x7f341019f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:17.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.619+0000 7f3415000640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3410076970 0x7f341019f600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f33f800e9f0 tx=0x7f33f800edf0 comp rx=0 tx=0).stop 2024-04-26T20:08:17.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.620+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 shutdown_connections 2024-04-26T20:08:17.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.620+0000 7f3415000640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33f40a3c30 0x7f33f40a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:17.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.620+0000 7f3415000640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f341010ee90 0x7f34101a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:17.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.620+0000 7f3415000640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3410076970 0x7f341019f600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:17.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.620+0000 7f3415000640 1 --2- 172.21.15.94:0/1521407073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3410075390 0x7f341019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:17.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.620+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 >> 172.21.15.94:0/1521407073 conn(0x7f34100fde30 msgr2=0x7f3410102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:17.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.620+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 shutdown_connections 2024-04-26T20:08:17.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:16.621+0000 7f3415000640 1 -- 172.21.15.94:0/1521407073 wait complete. 2024-04-26T20:08:17.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T20:08:17.391 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:16 smithi204 ceph-mon[32618]: pgmap v314: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:17.396 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:16 smithi204 ceph-mon[32618]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:08:17.403 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:16 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1521407073' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T20:08:17.413 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:16 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1521407073' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T20:08:17.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:16 smithi204 ceph-mon[32618]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T20:08:17.429 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:16 smithi103 ceph-mon[32635]: pgmap v314: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:17.440 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:16 smithi103 ceph-mon[32635]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:08:17.450 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:16 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1521407073' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T20:08:17.460 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:16 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1521407073' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T20:08:17.468 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:16 smithi103 ceph-mon[32635]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T20:08:17.518 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:16 smithi094 ceph-mon[28050]: pgmap v314: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:17.524 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:16 smithi094 ceph-mon[28050]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:08:17.611 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:16 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1521407073' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T20:08:17.665 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:16 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1521407073' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T20:08:17.864 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:16 smithi094 ceph-mon[28050]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T20:08:18.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.140+0000 7f9a64600640 1 -- 172.21.15.94:0/2837849123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c076970 msgr2=0x7f9a5c074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:18.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.140+0000 7f9a64600640 1 --2- 172.21.15.94:0/2837849123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c076970 0x7f9a5c074dc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009900 tx=0x7f9a4c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:08:18.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.140+0000 7f9a64600640 1 -- 172.21.15.94:0/2837849123 shutdown_connections 2024-04-26T20:08:18.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.140+0000 7f9a64600640 1 --2- 172.21.15.94:0/2837849123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a5c10a830 0x7f9a5c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:18.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.140+0000 7f9a64600640 1 --2- 172.21.15.94:0/2837849123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a5c075390 0x7f9a5c075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:18.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.140+0000 7f9a64600640 1 --2- 172.21.15.94:0/2837849123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c076970 0x7f9a5c074dc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:18.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.140+0000 7f9a64600640 1 -- 172.21.15.94:0/2837849123 >> 172.21.15.94:0/2837849123 conn(0x7f9a5c0fdee0 msgr2=0x7f9a5c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:18.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.140+0000 7f9a64600640 1 -- 172.21.15.94:0/2837849123 shutdown_connections 2024-04-26T20:08:18.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.140+0000 7f9a64600640 1 -- 172.21.15.94:0/2837849123 wait complete. 2024-04-26T20:08:18.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.141+0000 7f9a64600640 1 Processor -- start 2024-04-26T20:08:18.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a64600640 1 -- start start 2024-04-26T20:08:18.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a64600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a5c075390 0x7f9a5c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:18.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a64600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c076970 0x7f9a5c19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:18.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a64600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a5c10a830 0x7f9a5c1a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:18.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a64600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c1106d0 con 0x7f9a5c076970 2024-04-26T20:08:18.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a64600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c110550 con 0x7f9a5c10a830 2024-04-26T20:08:18.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a64600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c110850 con 0x7f9a5c075390 2024-04-26T20:08:18.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a61200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c076970 0x7f9a5c19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a61c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a5c075390 0x7f9a5c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:18.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a61c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a5c075390 0x7f9a5c19ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:55250/0 (socket says 172.21.15.94:55250) 2024-04-26T20:08:18.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a61200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c076970 0x7f9a5c19b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59710/0 (socket says 172.21.15.94:59710) 2024-04-26T20:08:18.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a61c00640 1 -- 172.21.15.94:0/3417135355 learned_addr learned my addr 172.21.15.94:0/3417135355 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:18.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.142+0000 7f9a62600640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a5c10a830 0x7f9a5c1a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:18.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a61c00640 1 -- 172.21.15.94:0/3417135355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a5c10a830 msgr2=0x7f9a5c1a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:18.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a61c00640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a5c10a830 0x7f9a5c1a23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:18.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a61c00640 1 -- 172.21.15.94:0/3417135355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c076970 msgr2=0x7f9a5c19b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:18.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a61c00640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c076970 0x7f9a5c19b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:18.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a61c00640 1 -- 172.21.15.94:0/3417135355 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5c1a2a40 con 0x7f9a5c075390 2024-04-26T20:08:18.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a61c00640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a5c075390 0x7f9a5c19ae20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c005ab0 tx=0x7f9a4c003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:18.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a5a000640 1 -- 172.21.15.94:0/3417135355 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c03f070 con 0x7f9a5c075390 2024-04-26T20:08:18.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a5c1a2cd0 con 0x7f9a5c075390 2024-04-26T20:08:18.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a5a000640 1 -- 172.21.15.94:0/3417135355 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a4c031c50 con 0x7f9a5c075390 2024-04-26T20:08:18.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.143+0000 7f9a5a000640 1 -- 172.21.15.94:0/3417135355 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c035030 con 0x7f9a5c075390 2024-04-26T20:08:18.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.144+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a5c1a31c0 con 0x7f9a5c075390 2024-04-26T20:08:18.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.145+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5c109560 con 0x7f9a5c075390 2024-04-26T20:08:18.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.147+0000 7f9a5a000640 1 -- 172.21.15.94:0/3417135355 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9a4c042070 con 0x7f9a5c075390 2024-04-26T20:08:18.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.148+0000 7f9a5a000640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9a400a3bb0 0x7f9a400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:18.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.148+0000 7f9a5a000640 1 -- 172.21.15.94:0/3417135355 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7f9a4c0f1200 con 0x7f9a5c075390 2024-04-26T20:08:18.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.148+0000 7f9a61200640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9a400a3bb0 0x7f9a400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:18.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.149+0000 7f9a61200640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9a400a3bb0 0x7f9a400a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c19c330 tx=0x7f9a48009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:18.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.150+0000 7f9a5a000640 1 -- 172.21.15.94:0/3417135355 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9a4c0bdcf0 con 0x7f9a5c075390 2024-04-26T20:08:18.909 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:18 smithi204 ceph-mon[32618]: pgmap v317: 97 pgs: 11 creating+peering, 12 active+clean, 74 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:18.915 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:18 smithi204 ceph-mon[32618]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T20:08:18.922 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:18 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3417135355' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:08:18.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:18 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:08:18.937 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:18 smithi204 ceph-mon[32618]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T20:08:19.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.397+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9a5c0635a0 con 0x7f9a5c075390 2024-04-26T20:08:19.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.659+0000 7f9a5a000640 1 -- 172.21.15.94:0/3417135355 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f9a4c0bd5f0 con 0x7f9a5c075390 2024-04-26T20:08:19.163 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T20:08:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9a400a3bb0 msgr2=0x7f9a400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:19.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9a400a3bb0 0x7f9a400a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c19c330 tx=0x7f9a48009500 comp rx=0 tx=0).stop 2024-04-26T20:08:19.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a5c075390 msgr2=0x7f9a5c19ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:19.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a5c075390 0x7f9a5c19ae20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c005ab0 tx=0x7f9a4c003b90 comp rx=0 tx=0).stop 2024-04-26T20:08:19.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 shutdown_connections 2024-04-26T20:08:19.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9a400a3bb0 0x7f9a400a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:19.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a5c10a830 0x7f9a5c1a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:19.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c076970 0x7f9a5c19b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:19.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 --2- 172.21.15.94:0/3417135355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a5c075390 0x7f9a5c19ae20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:19.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.666+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 >> 172.21.15.94:0/3417135355 conn(0x7f9a5c0fdee0 msgr2=0x7f9a5c106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.667+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 shutdown_connections 2024-04-26T20:08:19.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:18.667+0000 7f9a64600640 1 -- 172.21.15.94:0/3417135355 wait complete. 2024-04-26T20:08:19.278 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-26T20:08:19.285 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-26T20:08:19.359 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:18 smithi094 ceph-mon[28050]: pgmap v317: 97 pgs: 11 creating+peering, 12 active+clean, 74 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:19.464 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:18 smithi094 ceph-mon[28050]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T20:08:19.548 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:18 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3417135355' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:08:19.560 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:18 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:08:19.569 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:18 smithi094 ceph-mon[28050]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T20:08:19.630 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:18 smithi103 ceph-mon[32635]: pgmap v317: 97 pgs: 11 creating+peering, 12 active+clean, 74 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:19.683 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:18 smithi103 ceph-mon[32635]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T20:08:19.688 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:18 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3417135355' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:08:19.693 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:18 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:08:19.700 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:18 smithi103 ceph-mon[32635]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T20:08:19.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.846+0000 7f3972800640 1 -- 172.21.15.94:0/213354402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c108f90 msgr2=0x7f396c1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:19.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.846+0000 7f3972800640 1 --2- 172.21.15.94:0/213354402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c108f90 0x7f396c1093b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f395c009900 tx=0x7f395c02a410 comp rx=0 tx=0).stop 2024-04-26T20:08:19.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.846+0000 7f3972800640 1 -- 172.21.15.94:0/213354402 shutdown_connections 2024-04-26T20:08:19.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.846+0000 7f3972800640 1 --2- 172.21.15.94:0/213354402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f396c10ab40 0x7f396c1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:19.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.846+0000 7f3972800640 1 --2- 172.21.15.94:0/213354402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f396c10a180 0x7f396c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:19.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.846+0000 7f3972800640 1 --2- 172.21.15.94:0/213354402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c108f90 0x7f396c1093b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:20.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.846+0000 7f3972800640 1 -- 172.21.15.94:0/213354402 >> 172.21.15.94:0/213354402 conn(0x7f396c077f30 msgr2=0x7f396c07a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:20.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.846+0000 7f3972800640 1 -- 172.21.15.94:0/213354402 shutdown_connections 2024-04-26T20:08:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.846+0000 7f3972800640 1 -- 172.21.15.94:0/213354402 wait complete. 2024-04-26T20:08:20.247 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:19 smithi103 ceph-mon[32635]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T20:08:20.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:19 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T20:08:20.324 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:19 smithi103 ceph-mon[32635]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T20:08:20.333 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:19 smithi103 ceph-mon[32635]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T20:08:20.339 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:19 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] up:active 2024-04-26T20:08:20.401 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:19 smithi103 ceph-mon[32635]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:08:20.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:19 smithi204 ceph-mon[32618]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T20:08:20.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:19 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T20:08:20.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:19 smithi204 ceph-mon[32618]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T20:08:20.455 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:19 smithi204 ceph-mon[32618]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T20:08:20.461 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:19 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] up:active 2024-04-26T20:08:20.467 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:19 smithi204 ceph-mon[32618]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:08:20.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.847+0000 7f3972800640 1 Processor -- start 2024-04-26T20:08:20.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.847+0000 7f3972800640 1 -- start start 2024-04-26T20:08:20.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.847+0000 7f3972800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f396c108f90 0x7f396c19f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:20.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.847+0000 7f3972800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f396c10a180 0x7f396c19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:20.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.847+0000 7f3972800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c10ab40 0x7f396c1a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:20.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.847+0000 7f3972800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f396c114ad0 con 0x7f396c10ab40 2024-04-26T20:08:20.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.847+0000 7f3972800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f396c114950 con 0x7f396c108f90 2024-04-26T20:08:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.847+0000 7f3972800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f396c114c50 con 0x7f396c10a180 2024-04-26T20:08:20.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396be00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c10ab40 0x7f396c1a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:20.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396be00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c10ab40 0x7f396c1a67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59720/0 (socket says 172.21.15.94:59720) 2024-04-26T20:08:20.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396be00640 1 -- 172.21.15.94:0/233829669 learned_addr learned my addr 172.21.15.94:0/233829669 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:20.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f3963e00640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f396c10a180 0x7f396c19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396b400640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f396c108f90 0x7f396c19f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:20.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396be00640 1 -- 172.21.15.94:0/233829669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f396c10a180 msgr2=0x7f396c19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:20.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396be00640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f396c10a180 0x7f396c19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:20.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396be00640 1 -- 172.21.15.94:0/233829669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f396c108f90 msgr2=0x7f396c19f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:20.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396be00640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f396c108f90 0x7f396c19f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396be00640 1 -- 172.21.15.94:0/233829669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c1a6e10 con 0x7f396c10ab40 2024-04-26T20:08:21.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.848+0000 7f396be00640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c10ab40 0x7f396c1a67a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f396400eb10 tx=0x7f396400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:21.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.849+0000 7f3968c00640 1 -- 172.21.15.94:0/233829669 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964004b20 con 0x7f396c10ab40 2024-04-26T20:08:21.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.849+0000 7f3968c00640 1 -- 172.21.15.94:0/233829669 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3964004cc0 con 0x7f396c10ab40 2024-04-26T20:08:21.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.849+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c1a7100 con 0x7f396c10ab40 2024-04-26T20:08:21.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.849+0000 7f3968c00640 1 -- 172.21.15.94:0/233829669 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964010670 con 0x7f396c10ab40 2024-04-26T20:08:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.849+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c1a7650 con 0x7f396c10ab40 2024-04-26T20:08:21.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.850+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c1103d0 con 0x7f396c10ab40 2024-04-26T20:08:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.852+0000 7f3968c00640 1 -- 172.21.15.94:0/233829669 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3964005000 con 0x7f396c10ab40 2024-04-26T20:08:21.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.853+0000 7f3968c00640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f39500a3bb0 0x7f39500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:21.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.854+0000 7f3968c00640 1 -- 172.21.15.94:0/233829669 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f39640d0a80 con 0x7f396c10ab40 2024-04-26T20:08:21.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.854+0000 7f396b400640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f39500a3bb0 0x7f39500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:21.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.854+0000 7f396b400640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f39500a3bb0 0x7f39500a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f395c009230 tx=0x7f395c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:21.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:19.857+0000 7f3968c00640 1 -- 172.21.15.94:0/233829669 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3964054060 con 0x7f396c10ab40 2024-04-26T20:08:21.514 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:19 smithi094 ceph-mon[28050]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T20:08:21.521 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:19 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T20:08:21.528 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:19 smithi094 ceph-mon[28050]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T20:08:21.534 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:19 smithi094 ceph-mon[28050]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T20:08:21.542 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:19 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] up:active 2024-04-26T20:08:21.616 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:19 smithi094 ceph-mon[28050]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:08:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.066+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f396c02dad0 con 0x7f396c10ab40 2024-04-26T20:08:21.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.628+0000 7f3968c00640 1 -- 172.21.15.94:0/233829669 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v87) ==== 157+0+0 (secure 0 0 0) 0x7f396409d510 con 0x7f396c10ab40 2024-04-26T20:08:21.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f39500a3bb0 msgr2=0x7f39500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:21.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f39500a3bb0 0x7f39500a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f395c009230 tx=0x7f395c0314a0 comp rx=0 tx=0).stop 2024-04-26T20:08:21.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c10ab40 msgr2=0x7f396c1a67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:21.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c10ab40 0x7f396c1a67a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f396400eb10 tx=0x7f396400c2d0 comp rx=0 tx=0).stop 2024-04-26T20:08:21.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 shutdown_connections 2024-04-26T20:08:21.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f39500a3bb0 0x7f39500a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:21.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f396c10ab40 0x7f396c1a67a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:21.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f396c10a180 0x7f396c19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:21.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 --2- 172.21.15.94:0/233829669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f396c108f90 0x7f396c19f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:21.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.633+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 >> 172.21.15.94:0/233829669 conn(0x7f396c077f30 msgr2=0x7f396c10d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.634+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 shutdown_connections 2024-04-26T20:08:21.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:20.634+0000 7f3972800640 1 -- 172.21.15.94:0/233829669 wait complete. 2024-04-26T20:08:21.953 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T20:08:22.027 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:20 smithi094 ceph-mon[28050]: pgmap v320: 97 pgs: 19 creating+peering, 42 active+clean, 36 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2024-04-26T20:08:22.124 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:20 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/233829669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T20:08:22.208 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:20 smithi094 ceph-mon[28050]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T20:08:22.221 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:20 smithi094 ceph-mon[28050]: Cluster is now healthy 2024-04-26T20:08:22.231 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:21 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/233829669' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T20:08:22.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:21 smithi094 ceph-mon[28050]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T20:08:22.307 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:20 smithi204 ceph-mon[32618]: pgmap v320: 97 pgs: 19 creating+peering, 42 active+clean, 36 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2024-04-26T20:08:22.315 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:20 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/233829669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T20:08:22.324 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:20 smithi204 ceph-mon[32618]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T20:08:22.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:20 smithi204 ceph-mon[32618]: Cluster is now healthy 2024-04-26T20:08:22.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:21 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/233829669' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T20:08:22.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:21 smithi204 ceph-mon[32618]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T20:08:22.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:20 smithi103 ceph-mon[32635]: pgmap v320: 97 pgs: 19 creating+peering, 42 active+clean, 36 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2024-04-26T20:08:22.456 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:20 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/233829669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T20:08:22.463 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:20 smithi103 ceph-mon[32635]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T20:08:22.470 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:20 smithi103 ceph-mon[32635]: Cluster is now healthy 2024-04-26T20:08:22.477 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:21 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/233829669' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T20:08:22.485 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:21 smithi103 ceph-mon[32635]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T20:08:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.523+0000 7f81d3200640 1 -- 172.21.15.94:0/3072743992 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81cc075390 msgr2=0x7f81cc075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:22.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.523+0000 7f81d3200640 1 --2- 172.21.15.94:0/3072743992 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81cc075390 0x7f81cc075810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009930 tx=0x7f81c002a420 comp rx=0 tx=0).stop 2024-04-26T20:08:22.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.524+0000 7f81d3200640 1 -- 172.21.15.94:0/3072743992 shutdown_connections 2024-04-26T20:08:22.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.524+0000 7f81d3200640 1 --2- 172.21.15.94:0/3072743992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc10eda0 0x7f81cc111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.524+0000 7f81d3200640 1 --2- 172.21.15.94:0/3072743992 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81cc075390 0x7f81cc075810 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:22.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.524+0000 7f81d3200640 1 --2- 172.21.15.94:0/3072743992 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81cc076970 0x7f81cc074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:22.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.524+0000 7f81d3200640 1 -- 172.21.15.94:0/3072743992 >> 172.21.15.94:0/3072743992 conn(0x7f81cc0fdd40 msgr2=0x7f81cc100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:22.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.524+0000 7f81d3200640 1 -- 172.21.15.94:0/3072743992 shutdown_connections 2024-04-26T20:08:22.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.524+0000 7f81d3200640 1 -- 172.21.15.94:0/3072743992 wait complete. 2024-04-26T20:08:22.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.525+0000 7f81d3200640 1 Processor -- start 2024-04-26T20:08:22.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.525+0000 7f81d3200640 1 -- start start 2024-04-26T20:08:22.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.525+0000 7f81d3200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81cc075390 0x7f81cc19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:22.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.525+0000 7f81d3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81cc076970 0x7f81cc19f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:22.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.525+0000 7f81d3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc10eda0 0x7f81cc1a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:22.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.525+0000 7f81d3200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1148f0 con 0x7f81cc075390 2024-04-26T20:08:22.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.525+0000 7f81d3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc114770 con 0x7f81cc10eda0 2024-04-26T20:08:22.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.525+0000 7f81d3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc114a70 con 0x7f81cc076970 2024-04-26T20:08:22.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d1400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81cc076970 0x7f81cc19f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:23.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc10eda0 0x7f81cc1a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d1e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81cc075390 0x7f81cc19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:23.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc10eda0 0x7f81cc1a65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:46734/0 (socket says 172.21.15.94:46734) 2024-04-26T20:08:23.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d2800640 1 -- 172.21.15.94:0/1477198562 learned_addr learned my addr 172.21.15.94:0/1477198562 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:23.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d1400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81cc076970 0x7f81cc19f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:46406/0 (socket says 172.21.15.94:46406) 2024-04-26T20:08:23.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d2800640 1 -- 172.21.15.94:0/1477198562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81cc076970 msgr2=0x7f81cc19f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:23.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d2800640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81cc076970 0x7f81cc19f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:23.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d2800640 1 -- 172.21.15.94:0/1477198562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81cc075390 msgr2=0x7f81cc19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:23.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d2800640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81cc075390 0x7f81cc19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:23.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d2800640 1 -- 172.21.15.94:0/1477198562 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81cc1a6c50 con 0x7f81cc10eda0 2024-04-26T20:08:23.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.526+0000 7f81d2800640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc10eda0 0x7f81cc1a65e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f81c800eb10 tx=0x7f81c800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:23.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.527+0000 7f81c6a00640 1 -- 172.21.15.94:0/1477198562 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8004b20 con 0x7f81cc10eda0 2024-04-26T20:08:23.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.527+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81cc1a6f40 con 0x7f81cc10eda0 2024-04-26T20:08:23.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.527+0000 7f81c6a00640 1 -- 172.21.15.94:0/1477198562 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81c8004cc0 con 0x7f81cc10eda0 2024-04-26T20:08:23.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.527+0000 7f81c6a00640 1 -- 172.21.15.94:0/1477198562 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8010670 con 0x7f81cc10eda0 2024-04-26T20:08:23.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.527+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81cc1a73b0 con 0x7f81cc10eda0 2024-04-26T20:08:23.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.528+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81cc104e60 con 0x7f81cc10eda0 2024-04-26T20:08:23.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.530+0000 7f81c6a00640 1 -- 172.21.15.94:0/1477198562 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f81c8005000 con 0x7f81cc10eda0 2024-04-26T20:08:23.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.531+0000 7f81c6a00640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f81b40a3bb0 0x7f81b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:23.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.532+0000 7f81c6a00640 1 -- 172.21.15.94:0/1477198562 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8277+0+0 (secure 0 0 0) 0x7f81c80d0c20 con 0x7f81cc10eda0 2024-04-26T20:08:23.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.532+0000 7f81d1e00640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f81b40a3bb0 0x7f81b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:23.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.532+0000 7f81d1e00640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f81b40a3bb0 0x7f81b40a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009bd0 tx=0x7f81b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:23.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.534+0000 7f81c6a00640 1 -- 172.21.15.94:0/1477198562 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f81c8054060 con 0x7f81cc10eda0 2024-04-26T20:08:23.502 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:22 smithi204 ceph-mon[32618]: pgmap v322: 97 pgs: 18 creating+peering, 71 active+clean, 8 unknown; 578 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 1 op/s 2024-04-26T20:08:23.511 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:22 smithi103 ceph-mon[32635]: pgmap v322: 97 pgs: 18 creating+peering, 71 active+clean, 8 unknown; 578 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 1 op/s 2024-04-26T20:08:23.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:22.738+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f81cc02dcd0 con 0x7f81cc10eda0 2024-04-26T20:08:23.526 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:22 smithi094 ceph-mon[28050]: pgmap v322: 97 pgs: 18 creating+peering, 71 active+clean, 8 unknown; 578 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 1 op/s 2024-04-26T20:08:23.607 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:08:23 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b[63233]: 2024-04-26T20:08:23.131+0000 7ff2bca85700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T20:08:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1477198562' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:08:24.001 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:23 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:08:24.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1477198562' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:08:24.039 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:23 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:08:24.049 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1477198562' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:08:24.163 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:23 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:08:24.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.668+0000 7f81c6a00640 1 -- 172.21.15.94:0/1477198562 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v89) ==== 228+0+0 (secure 0 0 0) 0x7f81c809d570 con 0x7f81cc10eda0 2024-04-26T20:08:24.690 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data_ec' created 2024-04-26T20:08:24.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.675+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f81b40a3bb0 msgr2=0x7f81b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:24.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.675+0000 7f81d3200640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f81b40a3bb0 0x7f81b40a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009bd0 tx=0x7f81b8005f90 comp rx=0 tx=0).stop 2024-04-26T20:08:24.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.675+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc10eda0 msgr2=0x7f81cc1a65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.675+0000 7f81d3200640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc10eda0 0x7f81cc1a65e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f81c800eb10 tx=0x7f81c800c2d0 comp rx=0 tx=0).stop 2024-04-26T20:08:24.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.677+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 shutdown_connections 2024-04-26T20:08:24.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.677+0000 7f81d3200640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f81b40a3bb0 0x7f81b40a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:24.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.677+0000 7f81d3200640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc10eda0 0x7f81cc1a65e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:24.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.677+0000 7f81d3200640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81cc076970 0x7f81cc19f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:24.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.677+0000 7f81d3200640 1 --2- 172.21.15.94:0/1477198562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81cc075390 0x7f81cc19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:24.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.677+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 >> 172.21.15.94:0/1477198562 conn(0x7f81cc0fdd40 msgr2=0x7f81cc102030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:24.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.677+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 shutdown_connections 2024-04-26T20:08:24.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:24.678+0000 7f81d3200640 1 -- 172.21.15.94:0/1477198562 wait complete. 2024-04-26T20:08:24.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-26T20:08:24.991 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:24 smithi094 ceph-mon[28050]: pgmap v323: 97 pgs: 97 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-04-26T20:08:25.150 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:24 smithi094 ceph-mon[28050]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T20:08:25.247 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:24 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T20:08:25.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:24 smithi094 ceph-mon[28050]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T20:08:25.263 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:24 smithi204 ceph-mon[32618]: pgmap v323: 97 pgs: 97 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-04-26T20:08:25.271 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:24 smithi204 ceph-mon[32618]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T20:08:25.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:24 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T20:08:25.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:24 smithi204 ceph-mon[32618]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T20:08:25.351 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:24 smithi103 ceph-mon[32635]: pgmap v323: 97 pgs: 97 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-04-26T20:08:25.357 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:24 smithi103 ceph-mon[32635]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T20:08:25.401 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:24 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T20:08:25.409 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:24 smithi103 ceph-mon[32635]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T20:08:25.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.198+0000 7f5f5ee00640 1 -- 172.21.15.94:0/1068349448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810a180 msgr2=0x7f5f5810a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:25.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.198+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/1068349448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810a180 0x7f5f5810a600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f40009900 tx=0x7f5f4002a410 comp rx=0 tx=0).stop 2024-04-26T20:08:25.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.198+0000 7f5f5ee00640 1 -- 172.21.15.94:0/1068349448 shutdown_connections 2024-04-26T20:08:25.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.198+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/1068349448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5810ab40 0x7f5f581113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.198+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/1068349448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810a180 0x7f5f5810a600 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:25.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.198+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/1068349448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f58108f90 0x7f5f581093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:25.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.198+0000 7f5f5ee00640 1 -- 172.21.15.94:0/1068349448 >> 172.21.15.94:0/1068349448 conn(0x7f5f58077f30 msgr2=0x7f5f5807a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.198+0000 7f5f5ee00640 1 -- 172.21.15.94:0/1068349448 shutdown_connections 2024-04-26T20:08:25.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.198+0000 7f5f5ee00640 1 -- 172.21.15.94:0/1068349448 wait complete. 2024-04-26T20:08:25.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.199+0000 7f5f5ee00640 1 Processor -- start 2024-04-26T20:08:25.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.199+0000 7f5f5ee00640 1 -- start start 2024-04-26T20:08:25.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.199+0000 7f5f5ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f58108f90 0x7f5f5819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:25.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.199+0000 7f5f5ee00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5810a180 0x7f5f5819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.199+0000 7f5f5ee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810ab40 0x7f5f581a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:25.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.199+0000 7f5f5ee00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5f58114ad0 con 0x7f5f5810a180 2024-04-26T20:08:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.199+0000 7f5f5ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5f58114950 con 0x7f5f58108f90 2024-04-26T20:08:25.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.199+0000 7f5f5ee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5f58114c50 con 0x7f5f5810ab40 2024-04-26T20:08:25.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f5ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810ab40 0x7f5f581a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:25.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f5ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810ab40 0x7f5f581a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:46424/0 (socket says 172.21.15.94:46424) 2024-04-26T20:08:25.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f5ce00640 1 -- 172.21.15.94:0/107495964 learned_addr learned my addr 172.21.15.94:0/107495964 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:25.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f4fe00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5810a180 0x7f5f5819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:25.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f57e00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f58108f90 0x7f5f5819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:25.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f5ce00640 1 -- 172.21.15.94:0/107495964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f58108f90 msgr2=0x7f5f5819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:25.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f5ce00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f58108f90 0x7f5f5819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:25.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f5ce00640 1 -- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5810a180 msgr2=0x7f5f5819f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f5ce00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5810a180 0x7f5f5819f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:26.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f5ce00640 1 -- 172.21.15.94:0/107495964 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f581a6db0 con 0x7f5f5810ab40 2024-04-26T20:08:26.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.200+0000 7f5f5ce00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810ab40 0x7f5f581a6740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000eb10 tx=0x7f5f5000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:26.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.201+0000 7f5f55600640 1 -- 172.21.15.94:0/107495964 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50004b20 con 0x7f5f5810ab40 2024-04-26T20:08:26.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.201+0000 7f5f55600640 1 -- 172.21.15.94:0/107495964 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f50004cc0 con 0x7f5f5810ab40 2024-04-26T20:08:26.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.201+0000 7f5f55600640 1 -- 172.21.15.94:0/107495964 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50010670 con 0x7f5f5810ab40 2024-04-26T20:08:26.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.201+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f581a70a0 con 0x7f5f5810ab40 2024-04-26T20:08:26.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.201+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f581a7570 con 0x7f5f5810ab40 2024-04-26T20:08:26.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.202+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f58110380 con 0x7f5f5810ab40 2024-04-26T20:08:26.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.204+0000 7f5f55600640 1 -- 172.21.15.94:0/107495964 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5f50005020 con 0x7f5f5810ab40 2024-04-26T20:08:26.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.205+0000 7f5f55600640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5f3c0a3c30 0x7f5f3c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:26.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.205+0000 7f5f55600640 1 -- 172.21.15.94:0/107495964 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f5f500d0b10 con 0x7f5f5810ab40 2024-04-26T20:08:26.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.205+0000 7f5f57e00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5f3c0a3c30 0x7f5f3c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:26.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.206+0000 7f5f57e00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5f3c0a3c30 0x7f5f3c0a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5f48005fd0 tx=0x7f5f4800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:26.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.209+0000 7f5f55600640 1 -- 172.21.15.94:0/107495964 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5f5009d270 con 0x7f5f5810ab40 2024-04-26T20:08:26.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.412+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f5f5802dad0 con 0x7f5f5810ab40 2024-04-26T20:08:26.316 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:25 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/107495964' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T20:08:26.359 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:25 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T20:08:26.365 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:25 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T20:08:26.372 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:25 smithi204 ceph-mon[32618]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T20:08:26.420 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:25 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/107495964' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T20:08:26.426 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:25 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T20:08:26.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:25 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T20:08:26.445 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:25 smithi103 ceph-mon[32635]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T20:08:26.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.668+0000 7f5f55600640 1 -- 172.21.15.94:0/107495964 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v90) ==== 169+0+0 (secure 0 0 0) 0x7f5f5009cb70 con 0x7f5f5810ab40 2024-04-26T20:08:26.490 INFO:teuthology.orchestra.run.smithi094.stderr:set pool 4 allow_ec_overwrites to true 2024-04-26T20:08:26.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.673+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5f3c0a3c30 msgr2=0x7f5f3c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:26.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.673+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5f3c0a3c30 0x7f5f3c0a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5f48005fd0 tx=0x7f5f4800b410 comp rx=0 tx=0).stop 2024-04-26T20:08:26.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.673+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810ab40 msgr2=0x7f5f581a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:26.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.673+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810ab40 0x7f5f581a6740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000eb10 tx=0x7f5f5000c2d0 comp rx=0 tx=0).stop 2024-04-26T20:08:26.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.674+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 shutdown_connections 2024-04-26T20:08:26.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.675+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5f3c0a3c30 0x7f5f3c0a60f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:26.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.675+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5810ab40 0x7f5f581a6740 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:26.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.675+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f5810a180 0x7f5f5819f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:26.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.675+0000 7f5f5ee00640 1 --2- 172.21.15.94:0/107495964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f58108f90 0x7f5f5819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:26.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.675+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 >> 172.21.15.94:0/107495964 conn(0x7f5f58077f30 msgr2=0x7f5f5810d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:26.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.675+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 shutdown_connections 2024-04-26T20:08:26.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:25.676+0000 7f5f5ee00640 1 -- 172.21.15.94:0/107495964 wait complete. 2024-04-26T20:08:26.785 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-26T20:08:26.793 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:25 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/107495964' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T20:08:26.802 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:25 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T20:08:26.810 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:25 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T20:08:26.885 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:25 smithi094 ceph-mon[28050]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T20:08:26.964 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:26 smithi094 ceph-mon[28050]: pgmap v326: 129 pgs: 6 creating+peering, 26 unknown, 97 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2024-04-26T20:08:26.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:26 smithi094 ceph-mon[28050]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:08:26.995 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:26 smithi094 ceph-mon[28050]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T20:08:27.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:26 smithi103 ceph-mon[32635]: pgmap v326: 129 pgs: 6 creating+peering, 26 unknown, 97 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2024-04-26T20:08:27.041 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:26 smithi103 ceph-mon[32635]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:08:27.048 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:26 smithi103 ceph-mon[32635]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T20:08:27.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.103+0000 7f82eae00640 1 -- 172.21.15.94:0/3258118756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 msgr2=0x7f82e410a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:27.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.103+0000 7f82eae00640 1 --2- 172.21.15.94:0/3258118756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 0x7f82e410a440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009900 tx=0x7f82d002a410 comp rx=0 tx=0).stop 2024-04-26T20:08:27.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.103+0000 7f82eae00640 1 -- 172.21.15.94:0/3258118756 shutdown_connections 2024-04-26T20:08:27.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.103+0000 7f82eae00640 1 --2- 172.21.15.94:0/3258118756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82e410a980 0x7f82e41112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:27.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.103+0000 7f82eae00640 1 --2- 172.21.15.94:0/3258118756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 0x7f82e410a440 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.103+0000 7f82eae00640 1 --2- 172.21.15.94:0/3258118756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e4108e70 0x7f82e4109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:27.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.103+0000 7f82eae00640 1 -- 172.21.15.94:0/3258118756 >> 172.21.15.94:0/3258118756 conn(0x7f82e4077e00 msgr2=0x7f82e407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:27.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.103+0000 7f82eae00640 1 -- 172.21.15.94:0/3258118756 shutdown_connections 2024-04-26T20:08:27.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.103+0000 7f82eae00640 1 -- 172.21.15.94:0/3258118756 wait complete. 2024-04-26T20:08:27.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.104+0000 7f82eae00640 1 Processor -- start 2024-04-26T20:08:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.104+0000 7f82eae00640 1 -- start start 2024-04-26T20:08:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82eae00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82e4108e70 0x7f82e419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:27.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82eae00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 0x7f82e419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:27.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82eae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e410a980 0x7f82e41a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82eae00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f82e41149f0 con 0x7f82e4109fc0 2024-04-26T20:08:27.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82eae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f82e4114870 con 0x7f82e410a980 2024-04-26T20:08:27.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82eae00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82e4114b70 con 0x7f82e4108e70 2024-04-26T20:08:27.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82e9a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82e4108e70 0x7f82e419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:27.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82e9000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 0x7f82e419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:27.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82ea400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e410a980 0x7f82e41a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:27.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82e9a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82e4108e70 0x7f82e419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:46432/0 (socket says 172.21.15.94:46432) 2024-04-26T20:08:27.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82e9a00640 1 -- 172.21.15.94:0/617134605 learned_addr learned my addr 172.21.15.94:0/617134605 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:27.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.105+0000 7f82e9000640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 0x7f82e419f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49010/0 (socket says 172.21.15.94:49010) 2024-04-26T20:08:27.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e9a00640 1 -- 172.21.15.94:0/617134605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e410a980 msgr2=0x7f82e41a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:27.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e9a00640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e410a980 0x7f82e41a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:27.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e9a00640 1 -- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 msgr2=0x7f82e419f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:27.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e9a00640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 0x7f82e419f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:27.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e9a00640 1 -- 172.21.15.94:0/617134605 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e41a6c80 con 0x7f82e4108e70 2024-04-26T20:08:27.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82ea400640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e410a980 0x7f82e41a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:27.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e9000640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 0x7f82e419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:27.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e9a00640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82e4108e70 0x7f82e419f060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f82d800e9f0 tx=0x7f82d800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:27.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e2000640 1 -- 172.21.15.94:0/617134605 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d800cd80 con 0x7f82e4108e70 2024-04-26T20:08:27.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e2000640 1 -- 172.21.15.94:0/617134605 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d80108e0 con 0x7f82e4108e70 2024-04-26T20:08:27.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e41a6f70 con 0x7f82e4108e70 2024-04-26T20:08:27.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82e2000640 1 -- 172.21.15.94:0/617134605 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8018980 con 0x7f82e4108e70 2024-04-26T20:08:27.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.106+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e41a73e0 con 0x7f82e4108e70 2024-04-26T20:08:27.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.107+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e4110180 con 0x7f82e4108e70 2024-04-26T20:08:27.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.110+0000 7f82e2000640 1 -- 172.21.15.94:0/617134605 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f82d8010430 con 0x7f82e4108e70 2024-04-26T20:08:27.940 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:26 smithi204 ceph-mon[32618]: pgmap v326: 129 pgs: 6 creating+peering, 26 unknown, 97 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2024-04-26T20:08:27.945 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:26 smithi204 ceph-mon[32618]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:08:28.020 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:26 smithi204 ceph-mon[32618]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T20:08:28.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.111+0000 7f82e2000640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f82cc0a3c30 0x7f82cc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:28.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.111+0000 7f82e2000640 1 -- 172.21.15.94:0/617134605 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8770+0+0 (secure 0 0 0) 0x7f82d80d0d80 con 0x7f82e4108e70 2024-04-26T20:08:28.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.111+0000 7f82e9000640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f82cc0a3c30 0x7f82cc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:28.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.112+0000 7f82e9000640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f82cc0a3c30 0x7f82cc0a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f82d0005ab0 tx=0x7f82d0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.113+0000 7f82e2000640 1 -- 172.21.15.94:0/617134605 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f82d809ce90 con 0x7f82e4108e70 2024-04-26T20:08:28.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.359+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f82e402dad0 con 0x7f82e4108e70 2024-04-26T20:08:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.714+0000 7f82e2000640 1 -- 172.21.15.94:0/617134605 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7f82d80a1190 con 0x7f82e4108e70 2024-04-26T20:08:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:added data pool 4 to fsmap 2024-04-26T20:08:28.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f82cc0a3c30 msgr2=0x7f82cc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:28.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f82cc0a3c30 0x7f82cc0a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f82d0005ab0 tx=0x7f82d0005a20 comp rx=0 tx=0).stop 2024-04-26T20:08:28.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82e4108e70 msgr2=0x7f82e419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:28.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82e4108e70 0x7f82e419f060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f82d800e9f0 tx=0x7f82d800eeb0 comp rx=0 tx=0).stop 2024-04-26T20:08:28.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 shutdown_connections 2024-04-26T20:08:28.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f82cc0a3c30 0x7f82cc0a60f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82e410a980 0x7f82e41a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:28.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82e4109fc0 0x7f82e419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:28.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 --2- 172.21.15.94:0/617134605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82e4108e70 0x7f82e419f060 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:28.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 >> 172.21.15.94:0/617134605 conn(0x7f82e4077e00 msgr2=0x7f82e410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:28.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 shutdown_connections 2024-04-26T20:08:28.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:27.718+0000 7f82eae00640 1 -- 172.21.15.94:0/617134605 wait complete. 2024-04-26T20:08:28.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:08:28.508 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:27 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/617134605' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T20:08:28.516 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:27 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T20:08:28.526 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:27 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/617134605' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T20:08:28.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:27 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T20:08:28.543 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:27 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/617134605' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T20:08:28.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:27 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T20:08:28.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.737+0000 7fb8a3000640 1 -- 172.21.15.94:0/1325601139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb89c106740 msgr2=0x7fb89c10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:28.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.737+0000 7fb8a3000640 1 --2- 172.21.15.94:0/1325601139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb89c106740 0x7fb89c10cfd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb894009ca0 tx=0x7fb89402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:08:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.737+0000 7fb8a3000640 1 -- 172.21.15.94:0/1325601139 shutdown_connections 2024-04-26T20:08:28.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.737+0000 7fb8a3000640 1 --2- 172.21.15.94:0/1325601139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb89c106740 0x7fb89c10cfd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:28.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.737+0000 7fb8a3000640 1 --2- 172.21.15.94:0/1325601139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb89c105d80 0x7fb89c106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:28.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.737+0000 7fb8a3000640 1 --2- 172.21.15.94:0/1325601139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89c104b90 0x7fb89c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:28.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.737+0000 7fb8a3000640 1 -- 172.21.15.94:0/1325601139 >> 172.21.15.94:0/1325601139 conn(0x7fb89c100120 msgr2=0x7fb89c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:28.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.737+0000 7fb8a3000640 1 -- 172.21.15.94:0/1325601139 shutdown_connections 2024-04-26T20:08:28.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.737+0000 7fb8a3000640 1 -- 172.21.15.94:0/1325601139 wait complete. 2024-04-26T20:08:28.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a3000640 1 Processor -- start 2024-04-26T20:08:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a3000640 1 -- start start 2024-04-26T20:08:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a3000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89c104b90 0x7fb89c19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:29.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a3000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb89c105d80 0x7fb89c19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:29.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a3000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb89c106740 0x7fb89c1a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:29.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a3000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c1106c0 con 0x7fb89c106740 2024-04-26T20:08:29.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a3000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c110540 con 0x7fb89c104b90 2024-04-26T20:08:29.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a3000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c110840 con 0x7fb89c105d80 2024-04-26T20:08:29.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a1000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb89c106740 0x7fb89c1a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:29.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a1000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb89c106740 0x7fb89c1a22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49038/0 (socket says 172.21.15.94:49038) 2024-04-26T20:08:29.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb8a1000640 1 -- 172.21.15.94:0/2582491031 learned_addr learned my addr 172.21.15.94:0/2582491031 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:29.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb89be00640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89c104b90 0x7fb89c19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:29.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.738+0000 7fb89b400640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb89c105d80 0x7fb89c19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:29.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb89be00640 1 -- 172.21.15.94:0/2582491031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb89c105d80 msgr2=0x7fb89c19b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:29.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb89be00640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb89c105d80 0x7fb89c19b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:29.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb89be00640 1 -- 172.21.15.94:0/2582491031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb89c106740 msgr2=0x7fb89c1a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:29.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb89be00640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb89c106740 0x7fb89c1a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:29.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb89be00640 1 -- 172.21.15.94:0/2582491031 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89c1a2960 con 0x7fb89c104b90 2024-04-26T20:08:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb89be00640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89c104b90 0x7fb89c19ad50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00e9c0 tx=0x7fb88c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:29.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb898c00640 1 -- 172.21.15.94:0/2582491031 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c00ccc0 con 0x7fb89c104b90 2024-04-26T20:08:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89c1a2c50 con 0x7fb89c104b90 2024-04-26T20:08:29.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb898c00640 1 -- 172.21.15.94:0/2582491031 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88c0035f0 con 0x7fb89c104b90 2024-04-26T20:08:29.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.739+0000 7fb898c00640 1 -- 172.21.15.94:0/2582491031 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c00ce60 con 0x7fb89c104b90 2024-04-26T20:08:29.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.740+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb89c1a31a0 con 0x7fb89c104b90 2024-04-26T20:08:29.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.741+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb89c10beb0 con 0x7fb89c104b90 2024-04-26T20:08:29.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.742+0000 7fb898c00640 1 -- 172.21.15.94:0/2582491031 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb88c004000 con 0x7fb89c104b90 2024-04-26T20:08:29.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.744+0000 7fb898c00640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8800a3bb0 0x7fb8800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:29.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.744+0000 7fb898c00640 1 -- 172.21.15.94:0/2582491031 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fb88c0d0b60 con 0x7fb89c104b90 2024-04-26T20:08:29.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.744+0000 7fb89b400640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8800a3bb0 0x7fb8800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:29.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.745+0000 7fb89b400640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8800a3bb0 0x7fb8800a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb888009bd0 tx=0x7fb888005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:29.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.748+0000 7fb898c00640 1 -- 172.21.15.94:0/2582491031 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb88c09d2a0 con 0x7fb89c104b90 2024-04-26T20:08:29.644 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:28 smithi204 ceph-mon[32618]: pgmap v329: 129 pgs: 6 creating+peering, 20 unknown, 103 active+clean; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:29.650 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:28 smithi204 ceph-mon[32618]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T20:08:29.659 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:28 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T20:08:29.686 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:28 smithi204 ceph-mon[32618]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:08:29.716 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:28 smithi103 ceph-mon[32635]: pgmap v329: 129 pgs: 6 creating+peering, 20 unknown, 103 active+clean; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:29.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:28 smithi103 ceph-mon[32635]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T20:08:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:28 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T20:08:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:28 smithi103 ceph-mon[32635]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:08:29.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:28 smithi094 ceph-mon[28050]: pgmap v329: 129 pgs: 6 creating+peering, 20 unknown, 103 active+clean; 579 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:29.793 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:28 smithi094 ceph-mon[28050]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T20:08:29.800 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:28 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T20:08:29.806 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:28 smithi094 ceph-mon[28050]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:08:29.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.994+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb89c02dad0 con 0x7fb89c104b90 2024-04-26T20:08:29.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.995+0000 7fb898c00640 1 -- 172.21.15.94:0/2582491031 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11806 (secure 0 0 0) 0x7fb88c09cba0 con 0x7fb89c104b90 2024-04-26T20:08:29.942 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:29.951 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:08:27.361985+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24575,"qdb_cluster":[24575]},"id":1}]} 2024-04-26T20:08:29.960 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 16 2024-04-26T20:08:30.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8800a3bb0 msgr2=0x7fb8800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:30.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8800a3bb0 0x7fb8800a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb888009bd0 tx=0x7fb888005f90 comp rx=0 tx=0).stop 2024-04-26T20:08:30.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89c104b90 msgr2=0x7fb89c19ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:30.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89c104b90 0x7fb89c19ad50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00e9c0 tx=0x7fb88c00ee80 comp rx=0 tx=0).stop 2024-04-26T20:08:30.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 shutdown_connections 2024-04-26T20:08:30.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8800a3bb0 0x7fb8800a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb89c106740 0x7fb89c1a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:30.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb89c105d80 0x7fb89c19b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:30.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 --2- 172.21.15.94:0/2582491031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb89c104b90 0x7fb89c19ad50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:30.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.998+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 >> 172.21.15.94:0/2582491031 conn(0x7fb89c100120 msgr2=0x7fb89c109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:30.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.999+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 shutdown_connections 2024-04-26T20:08:30.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:28.999+0000 7fb8a3000640 1 -- 172.21.15.94:0/2582491031 wait complete. 2024-04-26T20:08:30.407 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:08:30.413 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:29 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2582491031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:30.419 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:29 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2582491031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:30.425 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:29 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2582491031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:30.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.699+0000 7f524e400640 1 -- 172.21.15.94:0/2140894608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248104300 msgr2=0x7f524810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:30.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.699+0000 7f524e400640 1 --2- 172.21.15.94:0/2140894608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248104300 0x7f524810ab90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5240009be0 tx=0x7f524002a6c0 comp rx=0 tx=0).stop 2024-04-26T20:08:30.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.699+0000 7f524e400640 1 -- 172.21.15.94:0/2140894608 shutdown_connections 2024-04-26T20:08:30.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.699+0000 7f524e400640 1 --2- 172.21.15.94:0/2140894608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248104300 0x7f524810ab90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:30.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.699+0000 7f524e400640 1 --2- 172.21.15.94:0/2140894608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248103940 0x7f5248103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.699+0000 7f524e400640 1 --2- 172.21.15.94:0/2140894608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5248102750 0x7f5248102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.699+0000 7f524e400640 1 -- 172.21.15.94:0/2140894608 >> 172.21.15.94:0/2140894608 conn(0x7f52480fdcc0 msgr2=0x7f5248100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:30.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.699+0000 7f524e400640 1 -- 172.21.15.94:0/2140894608 shutdown_connections 2024-04-26T20:08:30.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.699+0000 7f524e400640 1 -- 172.21.15.94:0/2140894608 wait complete. 2024-04-26T20:08:30.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.700+0000 7f524e400640 1 Processor -- start 2024-04-26T20:08:30.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.700+0000 7f524e400640 1 -- start start 2024-04-26T20:08:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.700+0000 7f524e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248102750 0x7f52481988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:31.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524e400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248103940 0x7f5248198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:31.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524e400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5248104300 0x7f524819fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:31.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524e400640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f524810e260 con 0x7f5248104300 2024-04-26T20:08:31.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f524810e0e0 con 0x7f5248102750 2024-04-26T20:08:31.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524e400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f524810e3e0 con 0x7f5248103940 2024-04-26T20:08:31.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524da00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5248104300 0x7f524819fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:31.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f5247e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248103940 0x7f5248198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:31.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248102750 0x7f52481988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:31.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524da00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5248104300 0x7f524819fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49054/0 (socket says 172.21.15.94:49054) 2024-04-26T20:08:31.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524da00640 1 -- 172.21.15.94:0/1566309135 learned_addr learned my addr 172.21.15.94:0/1566309135 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:31.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524d000640 1 -- 172.21.15.94:0/1566309135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248103940 msgr2=0x7f5248198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:31.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524d000640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248103940 0x7f5248198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:31.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524d000640 1 -- 172.21.15.94:0/1566309135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5248104300 msgr2=0x7f524819fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:31.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524d000640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5248104300 0x7f524819fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:31.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.701+0000 7f524d000640 1 -- 172.21.15.94:0/1566309135 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52481a04f0 con 0x7f5248102750 2024-04-26T20:08:31.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.702+0000 7f524d000640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248102750 0x7f52481988d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f523c005fd0 tx=0x7f523c00ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:31.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.702+0000 7f5245600640 1 -- 172.21.15.94:0/1566309135 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c00ccc0 con 0x7f5248102750 2024-04-26T20:08:31.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.702+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52481a07e0 con 0x7f5248102750 2024-04-26T20:08:31.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.702+0000 7f5245600640 1 -- 172.21.15.94:0/1566309135 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f523c00ee70 con 0x7f5248102750 2024-04-26T20:08:31.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.702+0000 7f5245600640 1 -- 172.21.15.94:0/1566309135 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c003e10 con 0x7f5248102750 2024-04-26T20:08:31.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.702+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52481a0d80 con 0x7f5248102750 2024-04-26T20:08:31.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.703+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5248109a70 con 0x7f5248102750 2024-04-26T20:08:31.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.705+0000 7f5245600640 1 -- 172.21.15.94:0/1566309135 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f523c003fb0 con 0x7f5248102750 2024-04-26T20:08:31.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.707+0000 7f5245600640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52300a3bb0 0x7f52300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:31.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.707+0000 7f5245600640 1 -- 172.21.15.94:0/1566309135 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f523c0d0910 con 0x7f5248102750 2024-04-26T20:08:31.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.707+0000 7f5247e00640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52300a3bb0 0x7f52300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:31.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.708+0000 7f5247e00640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52300a3bb0 0x7f52300a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5248199de0 tx=0x7f5238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:31.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.709+0000 7f5245600640 1 -- 172.21.15.94:0/1566309135 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f523c09d0e0 con 0x7f5248102750 2024-04-26T20:08:31.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.914+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f524802dc70 con 0x7f5248102750 2024-04-26T20:08:31.781 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:30 smithi204 ceph-mon[32618]: pgmap v331: 129 pgs: 6 creating+peering, 123 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:31.787 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:30 smithi204 ceph-mon[32618]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:08:31.793 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:30 smithi204 ceph-mon[32618]: Cluster is now healthy 2024-04-26T20:08:31.800 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:30 smithi103 ceph-mon[32635]: pgmap v331: 129 pgs: 6 creating+peering, 123 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:31.807 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:30 smithi103 ceph-mon[32635]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:08:31.874 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:30 smithi103 ceph-mon[32635]: Cluster is now healthy 2024-04-26T20:08:31.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.916+0000 7f5245600640 1 -- 172.21.15.94:0/1566309135 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24165 (secure 0 0 0) 0x7f523c09c9e0 con 0x7f5248102750 2024-04-26T20:08:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:31.958 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":92,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:08:27.361832+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:08:31.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52300a3bb0 msgr2=0x7f52300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:32.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52300a3bb0 0x7f52300a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5248199de0 tx=0x7f5238005f90 comp rx=0 tx=0).stop 2024-04-26T20:08:32.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248102750 msgr2=0x7f52481988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:32.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248102750 0x7f52481988d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f523c005fd0 tx=0x7f523c00ec20 comp rx=0 tx=0).stop 2024-04-26T20:08:32.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 shutdown_connections 2024-04-26T20:08:32.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52300a3bb0 0x7f52300a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:32.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5248104300 0x7f524819fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:32.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248103940 0x7f5248198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:32.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 --2- 172.21.15.94:0/1566309135 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248102750 0x7f52481988d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:32.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.918+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 >> 172.21.15.94:0/1566309135 conn(0x7f52480fdcc0 msgr2=0x7f5248106c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:32.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.919+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 shutdown_connections 2024-04-26T20:08:32.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:30.919+0000 7f524e400640 1 -- 172.21.15.94:0/1566309135 wait complete. 2024-04-26T20:08:32.260 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:30 smithi094 ceph-mon[28050]: pgmap v331: 129 pgs: 6 creating+peering, 123 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:32.366 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:30 smithi094 ceph-mon[28050]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:08:32.430 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:30 smithi094 ceph-mon[28050]: Cluster is now healthy 2024-04-26T20:08:32.438 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:08:32.445 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:31 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1566309135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:32.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:31 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1566309135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:32.511 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:31 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1566309135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:32.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.658+0000 7f8a8be00640 1 -- 172.21.15.94:0/2667291296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c104360 msgr2=0x7f8a8c10abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:32.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.658+0000 7f8a8be00640 1 --2- 172.21.15.94:0/2667291296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c104360 0x7f8a8c10abf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8a84009900 tx=0x7f8a8402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:08:32.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.658+0000 7f8a8be00640 1 -- 172.21.15.94:0/2667291296 shutdown_connections 2024-04-26T20:08:32.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.658+0000 7f8a8be00640 1 --2- 172.21.15.94:0/2667291296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c104360 0x7f8a8c10abf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:32.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.658+0000 7f8a8be00640 1 --2- 172.21.15.94:0/2667291296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a8c1039a0 0x7f8a8c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:32.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.658+0000 7f8a8be00640 1 --2- 172.21.15.94:0/2667291296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a8c1027b0 0x7f8a8c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:32.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.658+0000 7f8a8be00640 1 -- 172.21.15.94:0/2667291296 >> 172.21.15.94:0/2667291296 conn(0x7f8a8c0fdd40 msgr2=0x7f8a8c100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:32.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.658+0000 7f8a8be00640 1 -- 172.21.15.94:0/2667291296 shutdown_connections 2024-04-26T20:08:32.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.658+0000 7f8a8be00640 1 -- 172.21.15.94:0/2667291296 wait complete. 2024-04-26T20:08:32.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.659+0000 7f8a8be00640 1 Processor -- start 2024-04-26T20:08:32.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.659+0000 7f8a8be00640 1 -- start start 2024-04-26T20:08:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.659+0000 7f8a8be00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c1027b0 0x7f8a8c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.659+0000 7f8a8be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a8c1039a0 0x7f8a8c19b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:32.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.659+0000 7f8a8be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a8c104360 0x7f8a8c1a20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:32.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.659+0000 7f8a8be00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c10e310 con 0x7f8a8c1027b0 2024-04-26T20:08:32.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.659+0000 7f8a8be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c10e190 con 0x7f8a8c104360 2024-04-26T20:08:33.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.659+0000 7f8a8be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c10e490 con 0x7f8a8c1039a0 2024-04-26T20:08:33.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8aa00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c1027b0 0x7f8a8c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:33.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8aa00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c1027b0 0x7f8a8c19aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42058/0 (socket says 172.21.15.94:42058) 2024-04-26T20:08:33.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8aa00640 1 -- 172.21.15.94:0/3893066785 learned_addr learned my addr 172.21.15.94:0/3893066785 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:33.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a82000640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a8c1039a0 0x7f8a8c19b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:33.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8b400640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a8c104360 0x7f8a8c1a20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:33.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8aa00640 1 -- 172.21.15.94:0/3893066785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a8c1039a0 msgr2=0x7f8a8c19b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:33.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8aa00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a8c1039a0 0x7f8a8c19b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8aa00640 1 -- 172.21.15.94:0/3893066785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a8c104360 msgr2=0x7f8a8c1a20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:33.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8aa00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a8c104360 0x7f8a8c1a20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:33.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8aa00640 1 -- 172.21.15.94:0/3893066785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a8c1a2710 con 0x7f8a8c1027b0 2024-04-26T20:08:33.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.660+0000 7f8a8aa00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c1027b0 0x7f8a8c19aaf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00ebc0 tx=0x7f8a7c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:33.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.661+0000 7f8a83e00640 1 -- 172.21.15.94:0/3893066785 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c003b70 con 0x7f8a8c1027b0 2024-04-26T20:08:33.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.661+0000 7f8a83e00640 1 -- 172.21.15.94:0/3893066785 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7c003d10 con 0x7f8a8c1027b0 2024-04-26T20:08:33.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.661+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8c1a2a00 con 0x7f8a8c1027b0 2024-04-26T20:08:33.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.661+0000 7f8a83e00640 1 -- 172.21.15.94:0/3893066785 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c019800 con 0x7f8a8c1027b0 2024-04-26T20:08:33.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.661+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a8c1a2f50 con 0x7f8a8c1027b0 2024-04-26T20:08:33.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.662+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a8c1a31e0 con 0x7f8a8c1027b0 2024-04-26T20:08:33.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.664+0000 7f8a83e00640 1 -- 172.21.15.94:0/3893066785 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8a7c0199a0 con 0x7f8a8c1027b0 2024-04-26T20:08:33.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.666+0000 7f8a83e00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8a700a3bb0 0x7f8a700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:33.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.666+0000 7f8a83e00640 1 -- 172.21.15.94:0/3893066785 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f8a7c0d0fd0 con 0x7f8a8c1027b0 2024-04-26T20:08:33.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.666+0000 7f8a82000640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8a700a3bb0 0x7f8a700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:33.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.667+0000 7f8a82000640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8a700a3bb0 0x7f8a700a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c19c000 tx=0x7f8a74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.669+0000 7f8a83e00640 1 -- 172.21.15.94:0/3893066785 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8a7c09d7a0 con 0x7f8a8c1027b0 2024-04-26T20:08:33.615 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:32 smithi204 ceph-mon[32618]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:33.631 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:32 smithi103 ceph-mon[32635]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:33.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.873+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a8c1a33b0 con 0x7f8a8c1027b0 2024-04-26T20:08:33.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.877+0000 7f8a83e00640 1 -- 172.21.15.94:0/3893066785 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24165 (secure 0 0 0) 0x7f8a7c09d0a0 con 0x7f8a8c1027b0 2024-04-26T20:08:33.651 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:33.696 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":92,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:08:27.361832+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:08:33.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8a700a3bb0 msgr2=0x7f8a700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:33.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8a700a3bb0 0x7f8a700a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c19c000 tx=0x7f8a74005f90 comp rx=0 tx=0).stop 2024-04-26T20:08:33.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c1027b0 msgr2=0x7f8a8c19aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:33.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c1027b0 0x7f8a8c19aaf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00ebc0 tx=0x7f8a7c00c6a0 comp rx=0 tx=0).stop 2024-04-26T20:08:33.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 shutdown_connections 2024-04-26T20:08:33.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8a700a3bb0 0x7f8a700a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:33.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a8c104360 0x7f8a8c1a20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a8c1039a0 0x7f8a8c19b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:33.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 --2- 172.21.15.94:0/3893066785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a8c1027b0 0x7f8a8c19aaf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:33.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 >> 172.21.15.94:0/3893066785 conn(0x7f8a8c0fdd40 msgr2=0x7f8a8c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:33.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 shutdown_connections 2024-04-26T20:08:33.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:32.880+0000 7f8a8be00640 1 -- 172.21.15.94:0/3893066785 wait complete. 2024-04-26T20:08:33.966 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:32 smithi094 ceph-mon[28050]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:33.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:08:33.998 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:33 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3893066785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:34.008 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:33 smithi094 ceph-mon[28050]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:34.060 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:33 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3893066785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:34.153 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:33 smithi103 ceph-mon[32635]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:34.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:33 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3893066785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:34.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:33 smithi204 ceph-mon[32618]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:34.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.269+0000 7f946e600640 1 -- 172.21.15.94:0/2644006131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 msgr2=0x7f946810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:34.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.269+0000 7f946e600640 1 --2- 172.21.15.94:0/2644006131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 0x7f946810a400 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9454009900 tx=0x7f945402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:08:34.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.269+0000 7f946e600640 1 -- 172.21.15.94:0/2644006131 shutdown_connections 2024-04-26T20:08:34.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.269+0000 7f946e600640 1 --2- 172.21.15.94:0/2644006131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f946810a940 0x7f94681111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:34.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.269+0000 7f946e600640 1 --2- 172.21.15.94:0/2644006131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 0x7f946810a400 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:34.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.269+0000 7f946e600640 1 --2- 172.21.15.94:0/2644006131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9468108d90 0x7f94681091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.269+0000 7f946e600640 1 -- 172.21.15.94:0/2644006131 >> 172.21.15.94:0/2644006131 conn(0x7f9468077d30 msgr2=0x7f946807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.269+0000 7f946e600640 1 -- 172.21.15.94:0/2644006131 shutdown_connections 2024-04-26T20:08:34.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.269+0000 7f946e600640 1 -- 172.21.15.94:0/2644006131 wait complete. 2024-04-26T20:08:34.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.270+0000 7f946e600640 1 Processor -- start 2024-04-26T20:08:34.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.270+0000 7f946e600640 1 -- start start 2024-04-26T20:08:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f946e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9468108d90 0x7f946819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:34.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f946e600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 0x7f946819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:34.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f946e600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f946810a940 0x7f94681a65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:34.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f946e600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f94681148c0 con 0x7f9468109f80 2024-04-26T20:08:34.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f946e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9468114740 con 0x7f9468108d90 2024-04-26T20:08:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f946e600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9468114a40 con 0x7f946810a940 2024-04-26T20:08:34.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f9467e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 0x7f946819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:34.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f9467e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 0x7f946819f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42088/0 (socket says 172.21.15.94:42088) 2024-04-26T20:08:34.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f9467e00640 1 -- 172.21.15.94:0/1978742380 learned_addr learned my addr 172.21.15.94:0/1978742380 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:34.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f946d200640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9468108d90 0x7f946819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:34.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f946dc00640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f946810a940 0x7f94681a65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:34.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f9467e00640 1 -- 172.21.15.94:0/1978742380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f946810a940 msgr2=0x7f94681a65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:34.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f9467e00640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f946810a940 0x7f94681a65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:34.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f9467e00640 1 -- 172.21.15.94:0/1978742380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9468108d90 msgr2=0x7f946819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:34.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f9467e00640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9468108d90 0x7f946819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:34.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.271+0000 7f9467e00640 1 -- 172.21.15.94:0/1978742380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94681a6c10 con 0x7f9468109f80 2024-04-26T20:08:34.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.272+0000 7f9467e00640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 0x7f946819f530 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9454005940 tx=0x7f9454004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:34.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.272+0000 7f9465600640 1 -- 172.21.15.94:0/1978742380 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945403f030 con 0x7f9468109f80 2024-04-26T20:08:34.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.272+0000 7f9465600640 1 -- 172.21.15.94:0/1978742380 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9454003cb0 con 0x7f9468109f80 2024-04-26T20:08:34.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.272+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94681a6ea0 con 0x7f9468109f80 2024-04-26T20:08:34.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.272+0000 7f9465600640 1 -- 172.21.15.94:0/1978742380 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9454035070 con 0x7f9468109f80 2024-04-26T20:08:34.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.272+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94681a7310 con 0x7f9468109f80 2024-04-26T20:08:34.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.273+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94681101a0 con 0x7f9468109f80 2024-04-26T20:08:34.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.275+0000 7f9465600640 1 -- 172.21.15.94:0/1978742380 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9454042030 con 0x7f9468109f80 2024-04-26T20:08:34.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.277+0000 7f9465600640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94500a3c30 0x7f94500a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.277+0000 7f9465600640 1 -- 172.21.15.94:0/1978742380 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f94540f0e50 con 0x7f9468109f80 2024-04-26T20:08:35.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.277+0000 7f946d200640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94500a3c30 0x7f94500a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.278+0000 7f946d200640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94500a3c30 0x7f94500a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f945c005fd0 tx=0x7f945c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.281+0000 7f9465600640 1 -- 172.21.15.94:0/1978742380 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9454003e60 con 0x7f9468109f80 2024-04-26T20:08:35.133 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:34 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1978742380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:35.140 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:34 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1978742380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:35.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.535+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9468063530 con 0x7f9468109f80 2024-04-26T20:08:35.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.537+0000 7f9465600640 1 -- 172.21.15.94:0/1978742380 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11806 (secure 0 0 0) 0x7f94540bcfa0 con 0x7f9468109f80 2024-04-26T20:08:35.166 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:35.174 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:08:27.361985+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24575,"qdb_cluster":[24575]},"id":1}]} 2024-04-26T20:08:35.182 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 16 2024-04-26T20:08:35.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.539+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94500a3c30 msgr2=0x7f94500a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:35.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.539+0000 7f946e600640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94500a3c30 0x7f94500a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f945c005fd0 tx=0x7f945c00b390 comp rx=0 tx=0).stop 2024-04-26T20:08:35.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.539+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 msgr2=0x7f946819f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:35.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.539+0000 7f946e600640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 0x7f946819f530 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9454005940 tx=0x7f9454004430 comp rx=0 tx=0).stop 2024-04-26T20:08:35.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.540+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 shutdown_connections 2024-04-26T20:08:35.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.540+0000 7f946e600640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94500a3c30 0x7f94500a60f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:35.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.540+0000 7f946e600640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f946810a940 0x7f94681a65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:35.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.540+0000 7f946e600640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9468109f80 0x7f946819f530 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:35.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.540+0000 7f946e600640 1 --2- 172.21.15.94:0/1978742380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9468108d90 0x7f946819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:35.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.540+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 >> 172.21.15.94:0/1978742380 conn(0x7f9468077d30 msgr2=0x7f946810d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:35.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.540+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 shutdown_connections 2024-04-26T20:08:35.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:34.540+0000 7f946e600640 1 -- 172.21.15.94:0/1978742380 wait complete. 2024-04-26T20:08:35.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:08:35.539 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:34 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1978742380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:08:35.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.751+0000 7f14a1000640 1 -- 172.21.15.94:0/1012252411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f149c105ba0 msgr2=0x7f149c106020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:35.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.751+0000 7f14a1000640 1 --2- 172.21.15.94:0/1012252411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f149c105ba0 0x7f149c106020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1488009900 tx=0x7f148802a410 comp rx=0 tx=0).stop 2024-04-26T20:08:35.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.751+0000 7f14a1000640 1 -- 172.21.15.94:0/1012252411 shutdown_connections 2024-04-26T20:08:35.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.751+0000 7f14a1000640 1 --2- 172.21.15.94:0/1012252411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f149c106560 0x7f149c10cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:35.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.751+0000 7f14a1000640 1 --2- 172.21.15.94:0/1012252411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f149c105ba0 0x7f149c106020 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:36.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.751+0000 7f14a1000640 1 --2- 172.21.15.94:0/1012252411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f149c1049b0 0x7f149c104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.751+0000 7f14a1000640 1 -- 172.21.15.94:0/1012252411 >> 172.21.15.94:0/1012252411 conn(0x7f149c0fff40 msgr2=0x7f149c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:36.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.751+0000 7f14a1000640 1 -- 172.21.15.94:0/1012252411 shutdown_connections 2024-04-26T20:08:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.751+0000 7f14a1000640 1 -- 172.21.15.94:0/1012252411 wait complete. 2024-04-26T20:08:36.101 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:35 smithi103 ceph-mon[32635]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:36.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.752+0000 7f14a1000640 1 Processor -- start 2024-04-26T20:08:36.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.752+0000 7f14a1000640 1 -- start start 2024-04-26T20:08:36.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.752+0000 7f14a1000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f149c1049b0 0x7f149c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:36.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.752+0000 7f14a1000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f149c105ba0 0x7f149c19b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:36.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.752+0000 7f14a1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f149c106560 0x7f149c1a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f14a1000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f149c110530 con 0x7f149c1049b0 2024-04-26T20:08:36.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f14a1000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f149c1103b0 con 0x7f149c106560 2024-04-26T20:08:36.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f14a1000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f149c1106b0 con 0x7f149c105ba0 2024-04-26T20:08:36.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f149c1049b0 0x7f149c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f149c106560 0x7f149c1a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:36.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f149c105ba0 0x7f149c19b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:36.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f149c1049b0 0x7f149c19ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42110/0 (socket says 172.21.15.94:42110) 2024-04-26T20:08:36.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149b400640 1 -- 172.21.15.94:0/2911786601 learned_addr learned my addr 172.21.15.94:0/2911786601 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:36.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149be00640 1 -- 172.21.15.94:0/2911786601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f149c105ba0 msgr2=0x7f149c19b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149be00640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f149c105ba0 0x7f149c19b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:36.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149be00640 1 -- 172.21.15.94:0/2911786601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f149c1049b0 msgr2=0x7f149c19ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:36.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149be00640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f149c1049b0 0x7f149c19ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.753+0000 7f149be00640 1 -- 172.21.15.94:0/2911786601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149c1a2790 con 0x7f149c106560 2024-04-26T20:08:36.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.754+0000 7f149be00640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f149c106560 0x7f149c1a2120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1494009fd0 tx=0x7f149400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:36.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.754+0000 7f1493e00640 1 -- 172.21.15.94:0/2911786601 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149400ce20 con 0x7f149c106560 2024-04-26T20:08:36.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.754+0000 7f1493e00640 1 -- 172.21.15.94:0/2911786601 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1494005020 con 0x7f149c106560 2024-04-26T20:08:36.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.754+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149c1a2a80 con 0x7f149c106560 2024-04-26T20:08:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.754+0000 7f1493e00640 1 -- 172.21.15.94:0/2911786601 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1494010690 con 0x7f149c106560 2024-04-26T20:08:36.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.754+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f149c1a2f50 con 0x7f149c106560 2024-04-26T20:08:36.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.755+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f149c10bcd0 con 0x7f149c106560 2024-04-26T20:08:36.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.757+0000 7f1493e00640 1 -- 172.21.15.94:0/2911786601 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f14940054d0 con 0x7f149c106560 2024-04-26T20:08:36.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.758+0000 7f1493e00640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f14800a3bb0 0x7f14800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:36.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.758+0000 7f1493e00640 1 -- 172.21.15.94:0/2911786601 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f14940d0d60 con 0x7f149c106560 2024-04-26T20:08:37.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.758+0000 7f149b400640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f14800a3bb0 0x7f14800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:37.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.759+0000 7f149b400640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f14800a3bb0 0x7f14800a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f148c0058c0 tx=0x7f148c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:37.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.763+0000 7f1493e00640 1 -- 172.21.15.94:0/2911786601 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f149409d4a0 con 0x7f149c106560 2024-04-26T20:08:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.969+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f149c063530 con 0x7f149c106560 2024-04-26T20:08:37.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.971+0000 7f1493e00640 1 -- 172.21.15.94:0/2911786601 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24165 (secure 0 0 0) 0x7f149409cda0 con 0x7f149c106560 2024-04-26T20:08:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:37.205 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":92,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:08:27.361832+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:08:37.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f14800a3bb0 msgr2=0x7f14800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:37.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f14800a3bb0 0x7f14800a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f148c0058c0 tx=0x7f148c005f90 comp rx=0 tx=0).stop 2024-04-26T20:08:37.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f149c106560 msgr2=0x7f149c1a2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:37.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f149c106560 0x7f149c1a2120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1494009fd0 tx=0x7f149400ef10 comp rx=0 tx=0).stop 2024-04-26T20:08:37.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 shutdown_connections 2024-04-26T20:08:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f14800a3bb0 0x7f14800a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f149c106560 0x7f149c1a2120 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:37.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f149c105ba0 0x7f149c19b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:37.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 --2- 172.21.15.94:0/2911786601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f149c1049b0 0x7f149c19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:37.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 >> 172.21.15.94:0/2911786601 conn(0x7f149c0fff40 msgr2=0x7f149c108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:37.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 shutdown_connections 2024-04-26T20:08:37.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:35.975+0000 7f14a1000640 1 -- 172.21.15.94:0/2911786601 wait complete. 2024-04-26T20:08:37.445 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:35 smithi094 ceph-mon[28050]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:37.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T20:08:18.399846+0000', 'modified': '2024-04-26T20:08:27.361985+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.103:6839/1048742250', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 1048742250}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 1048742250}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24575, 'qdb_cluster': [24575]} 2024-04-26T20:08:37.512 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T20:08:37.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T20:08:37.614 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:36 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2911786601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:37.622 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:36 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2911786601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:37.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:35 smithi204 ceph-mon[32618]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:37.640 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:36 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2911786601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:08:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.835+0000 7fd870a00640 1 -- 172.21.15.94:0/584592970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd86c103960 msgr2=0x7fd86c103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:37.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.835+0000 7fd870a00640 1 --2- 172.21.15.94:0/584592970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd86c103960 0x7fd86c103de0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd854009900 tx=0x7fd85402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:08:37.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.835+0000 7fd870a00640 1 -- 172.21.15.94:0/584592970 shutdown_connections 2024-04-26T20:08:37.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.835+0000 7fd870a00640 1 --2- 172.21.15.94:0/584592970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd86c104320 0x7fd86c10abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.835+0000 7fd870a00640 1 --2- 172.21.15.94:0/584592970 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd86c103960 0x7fd86c103de0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:37.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.835+0000 7fd870a00640 1 --2- 172.21.15.94:0/584592970 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd86c102770 0x7fd86c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:37.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.835+0000 7fd870a00640 1 -- 172.21.15.94:0/584592970 >> 172.21.15.94:0/584592970 conn(0x7fd86c0fdce0 msgr2=0x7fd86c100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:38.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.835+0000 7fd870a00640 1 -- 172.21.15.94:0/584592970 shutdown_connections 2024-04-26T20:08:38.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.835+0000 7fd870a00640 1 -- 172.21.15.94:0/584592970 wait complete. 2024-04-26T20:08:38.052 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:37 smithi103 ceph-mon[32635]: pgmap v335: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:38.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.836+0000 7fd870a00640 1 Processor -- start 2024-04-26T20:08:38.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.836+0000 7fd870a00640 1 -- start start 2024-04-26T20:08:38.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd870a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd86c102770 0x7fd86c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd870a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd86c103960 0x7fd86c19b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd870a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd86c104320 0x7fd86c1a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:38.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd870a00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c10e2d0 con 0x7fd86c103960 2024-04-26T20:08:38.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd870a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c10e150 con 0x7fd86c102770 2024-04-26T20:08:38.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd870a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c10e450 con 0x7fd86c104320 2024-04-26T20:08:38.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd861e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd86c103960 0x7fd86c19b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:38.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd86a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd86c102770 0x7fd86c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:38.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd86a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd86c102770 0x7fd86c19ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:57750/0 (socket says 172.21.15.94:57750) 2024-04-26T20:08:38.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd86a800640 1 -- 172.21.15.94:0/1394476011 learned_addr learned my addr 172.21.15.94:0/1394476011 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:38.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd86b200640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd86c104320 0x7fd86c1a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:38.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd86a800640 1 -- 172.21.15.94:0/1394476011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd86c104320 msgr2=0x7fd86c1a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:38.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd86a800640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd86c104320 0x7fd86c1a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:38.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd86a800640 1 -- 172.21.15.94:0/1394476011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd86c103960 msgr2=0x7fd86c19b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:38.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd86a800640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd86c103960 0x7fd86c19b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:38.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.837+0000 7fd86a800640 1 -- 172.21.15.94:0/1394476011 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86c1a27b0 con 0x7fd86c102770 2024-04-26T20:08:38.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.838+0000 7fd86a800640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd86c102770 0x7fd86c19ab90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00ea80 tx=0x7fd85c00ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:38.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.838+0000 7fd863e00640 1 -- 172.21.15.94:0/1394476011 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00ce40 con 0x7fd86c102770 2024-04-26T20:08:38.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.838+0000 7fd863e00640 1 -- 172.21.15.94:0/1394476011 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd85c011920 con 0x7fd86c102770 2024-04-26T20:08:38.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.838+0000 7fd870a00640 1 -- 172.21.15.94:0/1394476011 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86c1a2aa0 con 0x7fd86c102770 2024-04-26T20:08:38.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.838+0000 7fd863e00640 1 -- 172.21.15.94:0/1394476011 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c01aaa0 con 0x7fd86c102770 2024-04-26T20:08:38.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.838+0000 7fd870a00640 1 -- 172.21.15.94:0/1394476011 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd86c1a2fc0 con 0x7fd86c102770 2024-04-26T20:08:38.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.841+0000 7fd863e00640 1 -- 172.21.15.94:0/1394476011 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd85c01ac40 con 0x7fd86c102770 2024-04-26T20:08:38.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.841+0000 7fd87426c740 1 Processor -- start 2024-04-26T20:08:38.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd87426c740 1 -- start start 2024-04-26T20:08:38.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd87426c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 0x55ef267e86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:38.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd87426c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267ed7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:38.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd87426c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:38.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd87426c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ef2674c600 con 0x55ef267e8cb0 2024-04-26T20:08:38.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd87426c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ef266e9100 con 0x55ef267e82c0 2024-04-26T20:08:38.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd87426c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ef267485c0 con 0x55ef267ede20 2024-04-26T20:08:38.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd86a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267ed7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:38.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd86a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267ed7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42136/0 (socket says 172.21.15.94:42136) 2024-04-26T20:08:38.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd86a800640 1 -- 172.21.15.94:0/801672994 learned_addr learned my addr 172.21.15.94:0/801672994 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:38.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd86b200640 1 --2- 172.21.15.94:0/801672994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 0x55ef267e86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:38.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.842+0000 7fd861e00640 1 --2- 172.21.15.94:0/801672994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:38.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd86a800640 1 -- 172.21.15.94:0/801672994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 msgr2=0x55ef267f02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:08:38.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd86a800640 1 --2- 172.21.15.94:0/801672994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:38.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd86a800640 1 -- 172.21.15.94:0/801672994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 msgr2=0x55ef267e86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:08:38.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd86a800640 1 --2- 172.21.15.94:0/801672994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 0x55ef267e86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:38.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd86a800640 1 -- 172.21.15.94:0/801672994 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef2645e2d0 con 0x55ef267e8cb0 2024-04-26T20:08:38.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd86a800640 1 --2- 172.21.15.94:0/801672994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267ed7b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0d4ce0 tx=0x7fd85c0d9600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fea4e749fe0ef36 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:38.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd863e00640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8500a3bb0 0x7fd8500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:38.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd85b400640 1 -- 172.21.15.94:0/801672994 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c098e90 con 0x55ef267e8cb0 2024-04-26T20:08:38.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd85b400640 1 -- 172.21.15.94:0/801672994 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd85c0e2070 con 0x55ef267e8cb0 2024-04-26T20:08:39.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd85b400640 1 -- 172.21.15.94:0/801672994 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c09c040 con 0x55ef267e8cb0 2024-04-26T20:08:39.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd863e00640 1 -- 172.21.15.94:0/1394476011 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fd85c0d0bb0 con 0x7fd86c102770 2024-04-26T20:08:39.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.843+0000 7fd86b200640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8500a3bb0 0x7fd8500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:39.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd87426c740 1 -- 172.21.15.94:0/801672994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 msgr2=0x55ef267ed7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:39.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd87426c740 1 --2- 172.21.15.94:0/801672994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267ed7b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0d4ce0 tx=0x7fd85c0d9600 comp rx=0 tx=0).stop 2024-04-26T20:08:39.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd86b200640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8500a3bb0 0x7fd8500a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55ef267e9510 tx=0x7fd864005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:39.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd87426c740 1 -- 172.21.15.94:0/801672994 shutdown_connections 2024-04-26T20:08:39.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd87426c740 1 --2- 172.21.15.94:0/801672994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:39.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd87426c740 1 --2- 172.21.15.94:0/801672994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267ed7b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:39.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd87426c740 1 --2- 172.21.15.94:0/801672994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 0x55ef267e86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:39.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd87426c740 1 -- 172.21.15.94:0/801672994 >> 172.21.15.94:0/801672994 conn(0x55ef2677cd10 msgr2=0x55ef267e3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:39.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd87426c740 1 -- 172.21.15.94:0/801672994 shutdown_connections 2024-04-26T20:08:39.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.844+0000 7fd87426c740 1 -- 172.21.15.94:0/801672994 wait complete. 2024-04-26T20:08:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.845+0000 7fd87426c740 1 -- start start 2024-04-26T20:08:39.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.845+0000 7fd87426c740 1 Processor -- start 2024-04-26T20:08:39.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.845+0000 7fd87426c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 0x55ef267f2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:39.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.845+0000 7fd87426c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:39.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.845+0000 7fd87426c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:39.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.845+0000 7fd87426c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ef267f0d70 con 0x55ef267e8cb0 2024-04-26T20:08:39.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.845+0000 7fd87426c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ef266d3f70 con 0x55ef267e82c0 2024-04-26T20:08:39.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.845+0000 7fd87426c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ef267f0ef0 con 0x55ef267ede20 2024-04-26T20:08:39.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.845+0000 7fd87426c740 10 client.0 _ll_register_callbacks cb 0x55ef26742480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:08:39.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd861e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd861e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267f2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42146/0 (socket says 172.21.15.94:42146) 2024-04-26T20:08:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd861e00640 1 -- 172.21.15.94:0/222388586 learned_addr learned my addr 172.21.15.94:0/222388586 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd86a800640 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 0x55ef267f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd86b200640 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd861e00640 1 -- 172.21.15.94:0/222388586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 msgr2=0x55ef267f9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:08:39.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd861e00640 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:39.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd861e00640 1 -- 172.21.15.94:0/222388586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 msgr2=0x55ef267f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:08:39.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd861e00640 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 0x55ef267f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd861e00640 1 -- 172.21.15.94:0/222388586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef265286b0 con 0x55ef267e8cb0 2024-04-26T20:08:39.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd86a800640 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 0x55ef267f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:39.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd861e00640 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267f2b90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd854006010 tx=0x7fd854004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a61be00a49635d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:39.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd84fe00640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-26T20:08:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd84fe00640 1 -- 172.21.15.94:0/222388586 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854035040 con 0x55ef267e8cb0 2024-04-26T20:08:39.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd84fe00640 1 -- 172.21.15.94:0/222388586 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fd854003ac0 con 0x55ef267e8cb0 2024-04-26T20:08:39.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.846+0000 7fd87426c740 1 -- 172.21.15.94:0/222388586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd85c0e2070 con 0x55ef267e8cb0 2024-04-26T20:08:39.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd84fe00640 1 -- 172.21.15.94:0/222388586 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fd854003cc0 con 0x55ef267e8cb0 2024-04-26T20:08:40.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd87426c740 10 client.14577 fetch_fsmap learned FSMap version 16 2024-04-26T20:08:40.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd87426c740 1 -- 172.21.15.94:0/222388586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55ef267fcbb0 con 0x55ef267e8cb0 2024-04-26T20:08:40.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd84fe00640 1 -- 172.21.15.94:0/222388586 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd85c0e2070 con 0x55ef267e8cb0 2024-04-26T20:08:40.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd84fe00640 1 -- 172.21.15.94:0/222388586 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fd85403c5d0 con 0x55ef267e8cb0 2024-04-26T20:08:40.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd87426c740 10 client.14577 fetch_fsmap finished waiting for FSMap version 16 2024-04-26T20:08:40.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd87426c740 10 client.14577 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:08:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd87426c740 20 client.14577 populate_metadata read hostname 'smithi094' 2024-04-26T20:08:40.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd87426c740 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x55ef267ff2c0 0x55ef26800730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:40.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd87426c740 4 client.14577 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:08:40.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.847+0000 7fd87426c740 1 -- 172.21.15.94:0/222388586 --> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef2675a210 con 0x55ef267ff2c0 2024-04-26T20:08:40.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.848+0000 7fd86a800640 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x55ef267ff2c0 0x55ef26800730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:40.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.848+0000 7fd86a800640 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x55ef267ff2c0 0x55ef26800730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e502027339d94b48 server_cookie=1dd367c34d356942 in_seq=0 out_seq=0 2024-04-26T20:08:40.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.848+0000 7fd84fe00640 10 client.14577 ms_handle_connect on v2:172.21.15.103:6838/1048742250 2024-04-26T20:08:40.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.849+0000 7fd84fe00640 1 -- 172.21.15.94:0/222388586 <== mds.0 v2:172.21.15.103:6838/1048742250 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef2675a210 con 0x55ef267ff2c0 2024-04-26T20:08:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.849+0000 7fd84fe00640 10 client.14577 handle_command_reply: tid=0 2024-04-26T20:08:40.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.849+0000 7fd87426c740 1 client.14577 shutdown 2024-04-26T20:08:40.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 -- 172.21.15.94:0/222388586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 msgr2=0x55ef267f2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:40.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267f2b90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd854006010 tx=0x7fd854004440 comp rx=0 tx=0).stop 2024-04-26T20:08:40.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 -- 172.21.15.94:0/222388586 shutdown_connections 2024-04-26T20:08:40.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x55ef267ff2c0 0x55ef26800730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:40.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e8cb0 0x55ef267f2b90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd84fe00640 0 client.14577 ms_handle_reset on v2:172.21.15.103:6838/1048742250 2024-04-26T20:08:40.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 --2- 172.21.15.94:0/222388586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e82c0 0x55ef267f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:40.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 -- 172.21.15.94:0/222388586 >> 172.21.15.94:0/222388586 conn(0x55ef2677cd10 msgr2=0x55ef267e2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:40.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 -- 172.21.15.94:0/222388586 shutdown_connections 2024-04-26T20:08:40.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 1 -- 172.21.15.94:0/222388586 wait complete. 2024-04-26T20:08:40.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.850+0000 7fd87426c740 20 client.14577 trim_cache size 0 max 16384 2024-04-26T20:08:40.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd87426c740 1 Processor -- start 2024-04-26T20:08:40.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd87426c740 1 -- start start 2024-04-26T20:08:40.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd87426c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267ea010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:40.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd87426c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267ea550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd87426c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:40.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd87426c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ef267fea70 con 0x55ef267e82c0 2024-04-26T20:08:40.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd87426c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ef267ff140 con 0x55ef267e8cb0 2024-04-26T20:08:40.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd87426c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ef267ff810 con 0x55ef267ede20 2024-04-26T20:08:40.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd86a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267ea010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:40.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd86a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267ea010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42156/0 (socket says 172.21.15.94:42156) 2024-04-26T20:08:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.870+0000 7fd86a800640 1 -- 172.21.15.94:0/1333532878 learned_addr learned my addr 172.21.15.94:0/1333532878 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:40.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd86b200640 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:40.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd861e00640 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267ea550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd86a800640 1 -- 172.21.15.94:0/1333532878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 msgr2=0x55ef267f9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:08:40.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd86a800640 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd86a800640 1 -- 172.21.15.94:0/1333532878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 msgr2=0x55ef267ea550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:08:40.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd86a800640 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267ea550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:40.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd86a800640 1 -- 172.21.15.94:0/1333532878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef267e9090 con 0x55ef267e82c0 2024-04-26T20:08:40.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd861e00640 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267ea550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd86a800640 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267ea010 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0d9b00 tx=0x7fd85c01a500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea43f2d47d1f01b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:41.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.871+0000 7fd85aa00640 1 -- 172.21.15.94:0/1333532878 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c09c370 con 0x55ef267e82c0 2024-04-26T20:08:41.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd85aa00640 1 -- 172.21.15.94:0/1333532878 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd85c0e2040 con 0x55ef267e82c0 2024-04-26T20:08:41.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd85aa00640 1 -- 172.21.15.94:0/1333532878 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c09c550 con 0x55ef267e82c0 2024-04-26T20:08:41.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd87426c740 1 -- 172.21.15.94:0/1333532878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 msgr2=0x55ef267ea010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:41.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd87426c740 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267ea010 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0d9b00 tx=0x7fd85c01a500 comp rx=0 tx=0).stop 2024-04-26T20:08:41.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd87426c740 1 -- 172.21.15.94:0/1333532878 shutdown_connections 2024-04-26T20:08:41.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd87426c740 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:41.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd87426c740 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267ea550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:41.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd87426c740 1 --2- 172.21.15.94:0/1333532878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267ea010 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd87426c740 1 -- 172.21.15.94:0/1333532878 >> 172.21.15.94:0/1333532878 conn(0x55ef2677cd10 msgr2=0x55ef267dedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:41.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd87426c740 1 -- 172.21.15.94:0/1333532878 shutdown_connections 2024-04-26T20:08:41.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.872+0000 7fd87426c740 1 -- 172.21.15.94:0/1333532878 wait complete. 2024-04-26T20:08:41.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.873+0000 7fd87426c740 1 -- start start 2024-04-26T20:08:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.873+0000 7fd87426c740 1 Processor -- start 2024-04-26T20:08:41.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.873+0000 7fd87426c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267f5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:41.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.873+0000 7fd87426c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267f5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:41.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.873+0000 7fd87426c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:41.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.873+0000 7fd87426c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55ef268005b0 con 0x55ef267e82c0 2024-04-26T20:08:41.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.873+0000 7fd87426c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ef267ffee0 con 0x55ef267e8cb0 2024-04-26T20:08:41.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.873+0000 7fd87426c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ef26800c80 con 0x55ef267ede20 2024-04-26T20:08:41.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.873+0000 7fd87426c740 10 client.0 _ll_register_callbacks cb 0x7fd854005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:08:41.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267f5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:41.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd861e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267f5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:41.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267f5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42158/0 (socket says 172.21.15.94:42158) 2024-04-26T20:08:41.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:41.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd861e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267f5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:57778/0 (socket says 172.21.15.94:57778) 2024-04-26T20:08:41.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86a800640 1 -- 172.21.15.94:0/3542537742 learned_addr learned my addr 172.21.15.94:0/3542537742 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:41.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86a800640 1 -- 172.21.15.94:0/3542537742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 msgr2=0x55ef267f86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:08:41.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86a800640 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:41.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86a800640 1 -- 172.21.15.94:0/3542537742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 msgr2=0x55ef267f5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:08:41.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86a800640 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267f5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:41.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86a800640 1 -- 172.21.15.94:0/3542537742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef267f8c90 con 0x55ef267e82c0 2024-04-26T20:08:42.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd861e00640 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267f5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:42.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd86a800640 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267f5140 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0d46d0 tx=0x7fd85c0d4ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c905f2b626f867f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:42.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd84fe00640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-26T20:08:42.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd84fe00640 1 -- 172.21.15.94:0/3542537742 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0b5020 con 0x55ef267e82c0 2024-04-26T20:08:42.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd84fe00640 1 -- 172.21.15.94:0/3542537742 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fd85c09aef0 con 0x55ef267e82c0 2024-04-26T20:08:42.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.874+0000 7fd87426c740 1 -- 172.21.15.94:0/3542537742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd85c0e2040 con 0x55ef267e82c0 2024-04-26T20:08:42.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd84fe00640 1 -- 172.21.15.94:0/3542537742 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fd85c09c730 con 0x55ef267e82c0 2024-04-26T20:08:42.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd87426c740 10 client.14583 fetch_fsmap learned FSMap version 16 2024-04-26T20:08:42.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd87426c740 1 -- 172.21.15.94:0/3542537742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55ef26865270 con 0x55ef267e82c0 2024-04-26T20:08:42.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd84fe00640 1 -- 172.21.15.94:0/3542537742 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd85c0e2040 con 0x55ef267e82c0 2024-04-26T20:08:42.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd84fe00640 1 -- 172.21.15.94:0/3542537742 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fd85c022030 con 0x55ef267e82c0 2024-04-26T20:08:42.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd87426c740 10 client.14583 fetch_fsmap finished waiting for FSMap version 16 2024-04-26T20:08:42.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd87426c740 10 client.14583 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:08:42.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd87426c740 20 client.14583 populate_metadata read hostname 'smithi094' 2024-04-26T20:08:42.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd87426c740 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x55ef26869500 0x55ef26866010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:42.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd87426c740 4 client.14583 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:08:42.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.875+0000 7fd87426c740 1 -- 172.21.15.94:0/3542537742 --> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] -- command(tid 0: {"prefix": "status"}) -- 0x55ef26754080 con 0x55ef26869500 2024-04-26T20:08:42.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd861e00640 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x55ef26869500 0x55ef26866010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:42.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd861e00640 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x55ef26869500 0x55ef26866010 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3aa47a231e3f405 server_cookie=80efb2546d8cd814 in_seq=0 out_seq=0 2024-04-26T20:08:42.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd84fe00640 10 client.14583 ms_handle_connect on v2:172.21.15.103:6838/1048742250 2024-04-26T20:08:42.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd84fe00640 1 -- 172.21.15.94:0/3542537742 <== mds.0 v2:172.21.15.103:6838/1048742250 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ef26754080 con 0x55ef26869500 2024-04-26T20:08:42.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd84fe00640 10 client.14583 handle_command_reply: tid=0 2024-04-26T20:08:42.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd87426c740 1 client.14583 shutdown 2024-04-26T20:08:42.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd87426c740 1 -- 172.21.15.94:0/3542537742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 msgr2=0x55ef267f5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:42.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd87426c740 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267f5140 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0d46d0 tx=0x7fd85c0d4ef0 comp rx=0 tx=0).stop 2024-04-26T20:08:42.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd87426c740 1 -- 172.21.15.94:0/3542537742 shutdown_connections 2024-04-26T20:08:42.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.876+0000 7fd87426c740 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x55ef26869500 0x55ef26866010 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:42.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd87426c740 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ef267ede20 0x55ef267f86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:42.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd87426c740 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ef267e8cb0 0x55ef267f5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:42.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd84fe00640 0 client.14583 ms_handle_reset on v2:172.21.15.103:6838/1048742250 2024-04-26T20:08:42.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd87426c740 1 --2- 172.21.15.94:0/3542537742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ef267e82c0 0x55ef267f5140 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:42.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd87426c740 1 -- 172.21.15.94:0/3542537742 >> 172.21.15.94:0/3542537742 conn(0x55ef2677cd10 msgr2=0x55ef267fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:42.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd87426c740 1 -- 172.21.15.94:0/3542537742 shutdown_connections 2024-04-26T20:08:42.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd87426c740 1 -- 172.21.15.94:0/3542537742 wait complete. 2024-04-26T20:08:43.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd87426c740 20 client.14583 trim_cache size 0 max 16384 2024-04-26T20:08:43.113 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-26T20:08:43.128 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T20:08:43.142 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-04-26T20:08:43.154 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24575, 2024-04-26T20:08:43.232 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-26T20:08:43.245 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-26T20:08:43.257 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-26T20:08:43.270 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 19.218110427999999, 2024-04-26T20:08:43.280 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 16, 2024-04-26T20:08:43.290 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 92, 2024-04-26T20:08:43.300 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 92, 2024-04-26T20:08:43.311 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 36.144614120999996 2024-04-26T20:08:43.324 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-26T20:08:43.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 -- 172.21.15.94:0/1394476011 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8500a3bb0 msgr2=0x7fd8500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:43.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8500a3bb0 0x7fd8500a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55ef267e9510 tx=0x7fd864005f90 comp rx=0 tx=0).stop 2024-04-26T20:08:43.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 -- 172.21.15.94:0/1394476011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd86c102770 msgr2=0x7fd86c19ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:43.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd86c102770 0x7fd86c19ab90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00ea80 tx=0x7fd85c00ef40 comp rx=0 tx=0).stop 2024-04-26T20:08:43.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 -- 172.21.15.94:0/1394476011 shutdown_connections 2024-04-26T20:08:43.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8500a3bb0 0x7fd8500a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:43.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd86c104320 0x7fd86c1a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:43.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd86c103960 0x7fd86c19b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:43.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 --2- 172.21.15.94:0/1394476011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd86c102770 0x7fd86c19ab90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:43.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.877+0000 7fd870a00640 1 -- 172.21.15.94:0/1394476011 >> 172.21.15.94:0/1394476011 conn(0x7fd86c0fdce0 msgr2=0x7fd86c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:43.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.878+0000 7fd870a00640 1 -- 172.21.15.94:0/1394476011 shutdown_connections 2024-04-26T20:08:43.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:37.878+0000 7fd870a00640 1 -- 172.21.15.94:0/1394476011 wait complete. 2024-04-26T20:08:43.687 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T20:08:43.706 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:08:43.719 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T20:08:43.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:37 smithi094 ceph-mon[28050]: pgmap v335: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:43.748 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:40 smithi094 ceph-mon[28050]: pgmap v336: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:43.759 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:42 smithi094 ceph-mon[28050]: pgmap v337: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:43.830 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:40 smithi103 ceph-mon[32635]: pgmap v336: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:43.844 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:42 smithi103 ceph-mon[32635]: pgmap v337: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:43.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:37 smithi204 ceph-mon[32618]: pgmap v335: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:44.046 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:40 smithi204 ceph-mon[32618]: pgmap v336: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:44.056 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:42 smithi204 ceph-mon[32618]: pgmap v337: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:08:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:44 smithi204 ceph-mon[32618]: pgmap v338: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:44.923 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:44 smithi094 ceph-mon[28050]: pgmap v338: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:45.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:44 smithi103 ceph-mon[32635]: pgmap v338: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:46.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.668+0000 7f416ca00700 1 -- 172.21.15.94:0/3922281961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f416810b630 msgr2=0x7f416810ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:46.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.668+0000 7f416ca00700 1 --2- 172.21.15.94:0/3922281961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f416810b630 0x7f416810ba10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f41580098f0 tx=0x7f4158009c00 comp rx=0 tx=0).stop 2024-04-26T20:08:46.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.668+0000 7f416ca00700 1 -- 172.21.15.94:0/3922281961 shutdown_connections 2024-04-26T20:08:46.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.668+0000 7f416ca00700 1 --2- 172.21.15.94:0/3922281961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4168105ff0 0x7f4168112500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:46.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.668+0000 7f416ca00700 1 --2- 172.21.15.94:0/3922281961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f416810b630 0x7f416810ba10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:46.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.668+0000 7f416ca00700 1 --2- 172.21.15.94:0/3922281961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4168105630 0x7f4168105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:46.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.668+0000 7f416ca00700 1 -- 172.21.15.94:0/3922281961 >> 172.21.15.94:0/3922281961 conn(0x7f4168101120 msgr2=0x7f4168103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.668+0000 7f416ca00700 1 -- 172.21.15.94:0/3922281961 shutdown_connections 2024-04-26T20:08:46.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.668+0000 7f416ca00700 1 -- 172.21.15.94:0/3922281961 wait complete. 2024-04-26T20:08:46.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.669+0000 7f416ca00700 1 Processor -- start 2024-04-26T20:08:46.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.669+0000 7f416ca00700 1 -- start start 2024-04-26T20:08:47.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f416ca00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4168105630 0x7f416819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f416ca00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4168105ff0 0x7f416819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:47.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f416ca00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f416810b630 0x7f41681a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:47.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f416ca00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4168115860 con 0x7f416810b630 2024-04-26T20:08:47.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f416ca00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f41681156e0 con 0x7f4168105ff0 2024-04-26T20:08:47.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f416ca00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f41681159e0 con 0x7f4168105630 2024-04-26T20:08:47.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f4166800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f416810b630 0x7f41681a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:47.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f4165e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4168105630 0x7f416819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:47.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f4166800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f416810b630 0x7f41681a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40564/0 (socket says 172.21.15.94:40564) 2024-04-26T20:08:47.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f4166800700 1 -- 172.21.15.94:0/3418653303 learned_addr learned my addr 172.21.15.94:0/3418653303 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:08:47.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.670+0000 7f4165400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4168105ff0 0x7f416819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:47.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f4165e00700 1 -- 172.21.15.94:0/3418653303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4168105ff0 msgr2=0x7f416819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:47.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f4165e00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4168105ff0 0x7f416819c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:47.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f4165e00700 1 -- 172.21.15.94:0/3418653303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f416810b630 msgr2=0x7f41681a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:47.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f4165e00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f416810b630 0x7f41681a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f4165e00700 1 -- 172.21.15.94:0/3418653303 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41681a0da0 con 0x7f4168105630 2024-04-26T20:08:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f4165400700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4168105ff0 0x7f416819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:08:47.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f4165e00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4168105630 0x7f416819bd90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f415800c010 tx=0x7f415800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:47.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f415ea00700 1 -- 172.21.15.94:0/3418653303 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580050c0 con 0x7f4168105630 2024-04-26T20:08:47.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41681a1030 con 0x7f4168105630 2024-04-26T20:08:47.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f415ea00700 1 -- 172.21.15.94:0/3418653303 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f415800b560 con 0x7f4168105630 2024-04-26T20:08:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.671+0000 7f415ea00700 1 -- 172.21.15.94:0/3418653303 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158003a00 con 0x7f4168105630 2024-04-26T20:08:47.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.672+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41681a8910 con 0x7f4168105630 2024-04-26T20:08:47.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.673+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f416806a110 con 0x7f4168105630 2024-04-26T20:08:47.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.675+0000 7f415ea00700 1 -- 172.21.15.94:0/3418653303 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f415800b710 con 0x7f4168105630 2024-04-26T20:08:47.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.676+0000 7f415ea00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f414c0a3c40 0x7f414c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:08:47.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.676+0000 7f415ea00700 1 -- 172.21.15.94:0/3418653303 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f41580d16e0 con 0x7f4168105630 2024-04-26T20:08:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.676+0000 7f4165400700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f414c0a3c40 0x7f414c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:08:47.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.677+0000 7f4165400700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f414c0a3c40 0x7f414c0a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f41681037c0 tx=0x7f4154009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:08:47.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.678+0000 7f415ea00700 1 -- 172.21.15.94:0/3418653303 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4158013030 con 0x7f4168105630 2024-04-26T20:08:47.717 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:46 smithi204 ceph-mon[32618]: pgmap v339: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:47.724 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:46 smithi103 ceph-mon[32635]: pgmap v339: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:47.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.954+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f416810dbe0 con 0x7f4168105630 2024-04-26T20:08:47.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.955+0000 7f415ea00700 1 -- 172.21.15.94:0/3418653303 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7f416810dbe0 con 0x7f4168105630 2024-04-26T20:08:47.887 INFO:teuthology.orchestra.run.smithi094.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T20:08:47.912 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:46 smithi094 ceph-mon[28050]: pgmap v339: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:47.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f414c0a3c40 msgr2=0x7f414c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:47.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f414c0a3c40 0x7f414c0a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f41681037c0 tx=0x7f4154009260 comp rx=0 tx=0).stop 2024-04-26T20:08:47.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4168105630 msgr2=0x7f416819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:47.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4168105630 0x7f416819bd90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f415800c010 tx=0x7f415800fa00 comp rx=0 tx=0).stop 2024-04-26T20:08:47.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 shutdown_connections 2024-04-26T20:08:48.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f416810b630 0x7f41681a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:48.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4168105630 0x7f416819bd90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:48.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f414c0a3c40 0x7f414c0a6100 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:48.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 --2- 172.21.15.94:0/3418653303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4168105ff0 0x7f416819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:48.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 >> 172.21.15.94:0/3418653303 conn(0x7f4168101120 msgr2=0x7f41681018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 shutdown_connections 2024-04-26T20:08:48.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:46.959+0000 7f416ca00700 1 -- 172.21.15.94:0/3418653303 wait complete. 2024-04-26T20:08:48.049 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T20:08:48.079 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T20:08:48.152 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi094.front.sepia.ceph.com 2024-04-26T20:08:48.238 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-26T20:08:48.247 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:08:48.254 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T20:08:48.263 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T20:08:48.325 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T20:08:48.390 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T20:08:48.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:48.401 DEBUG:teuthology.orchestra.run.smithi094:> ip addr 2024-04-26T20:08:48.409 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:47 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3418653303' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:08:48.416 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:47 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3418653303' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:08:48.422 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:47 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3418653303' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:08:48.487 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T20:08:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T20:08:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T20:08:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:08:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2024-04-26T20:08:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:08:48.671 INFO:teuthology.orchestra.run.smithi094.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:08:48.679 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4c brd ff:ff:ff:ff:ff:ff 2024-04-26T20:08:48.687 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f0 2024-04-26T20:08:48.714 INFO:teuthology.orchestra.run.smithi094.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:08:48.775 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4d brd ff:ff:ff:ff:ff:ff 2024-04-26T20:08:48.783 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f1 2024-04-26T20:08:48.791 INFO:teuthology.orchestra.run.smithi094.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:08:48.799 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:62 brd ff:ff:ff:ff:ff:ff 2024-04-26T20:08:48.807 INFO:teuthology.orchestra.run.smithi094.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T20:08:48.815 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:63 brd ff:ff:ff:ff:ff:ff 2024-04-26T20:08:48.863 INFO:teuthology.orchestra.run.smithi094.stdout: inet 172.21.15.94/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-26T20:08:48.871 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft 42044sec preferred_lft 42044sec 2024-04-26T20:08:48.879 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::ec4:7aff:fe8f:d763/64 scope link 2024-04-26T20:08:48.888 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:08:48.898 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T20:08:48.906 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:08:48.965 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:08:49.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add name ceph-brx type bridge 2024-04-26T20:08:49.040 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr flush dev ceph-brx 2024-04-26T20:08:49.048 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set ceph-brx up 2024-04-26T20:08:49.057 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T20:08:49.113 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:08:49.123 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:48 smithi103 ceph-mon[32635]: pgmap v340: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:49.130 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:48 smithi094 ceph-mon[28050]: pgmap v340: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:49.139 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:48 smithi204 ceph-mon[32618]: pgmap v340: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:08:49.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:08:49.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:49.666 DEBUG:teuthology.orchestra.run.smithi094:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T20:08:49.740 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-04-26T20:08:49.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:49.880 DEBUG:teuthology.orchestra.run.smithi094:> route 2024-04-26T20:08:49.958 INFO:teuthology.orchestra.run.smithi094.stdout:Kernel IP routing table 2024-04-26T20:08:50.042 INFO:teuthology.orchestra.run.smithi094.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T20:08:50.077 INFO:teuthology.orchestra.run.smithi094.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-26T20:08:50.085 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-26T20:08:50.093 INFO:teuthology.orchestra.run.smithi094.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T20:08:50.144 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:08:50.152 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:08:50.160 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-26T20:08:50.169 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-26T20:08:50.177 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-26T20:08:50.186 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:08:50.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:08:50.915 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:50 smithi094 ceph-mon[28050]: pgmap v341: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:50.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:08:50.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:50.968 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-04-26T20:08:51.040 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:50 smithi103 ceph-mon[32635]: pgmap v341: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:51.143 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:50 smithi204 ceph-mon[32618]: pgmap v341: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:51.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:51.252 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list-id 2024-04-26T20:08:51.340 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:08:51.348 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:08:51.418 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:08:51.423 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-26T20:08:51.450 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:08:51.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:08:51.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:08:51.649 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-26T20:08:51.657 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:08:51.664 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:08:51.740 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-26T20:08:51.748 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T20:08:51.761 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-26T20:08:51.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-26T20:08:51.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-26T20:08:51.930 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:08:52.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:08:52.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:08:52.296 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:08:52.373 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:08:52.417 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set brx.0 up 2024-04-26T20:08:52.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T20:08:52.507 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:08:52.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:08:52.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:08:52.740 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-26T20:08:52.778 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T20:08:52.790 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:08:52.886 INFO:teuthology.orchestra.run.smithi094.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T20:08:52.960 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T20:08:52.972 DEBUG:teuthology.orchestra.run.smithi094:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:08:53.074 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:52 smithi103 ceph-mon[32635]: pgmap v342: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:53.144 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:52 smithi094 ceph-mon[28050]: pgmap v342: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:53.207 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:52 smithi204 ceph-mon[32618]: pgmap v342: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:53.269 DEBUG:teuthology.orchestra.run.smithi094:> sudo modprobe fuse 2024-04-26T20:08:53.402 DEBUG:teuthology.orchestra.run.smithi094:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T20:08:53.477 INFO:teuthology.orchestra.run.smithi094.stdout:/proc 2024-04-26T20:08:53.489 INFO:teuthology.orchestra.run.smithi094.stdout:/sys 2024-04-26T20:08:53.503 INFO:teuthology.orchestra.run.smithi094.stdout:/dev 2024-04-26T20:08:53.515 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/security 2024-04-26T20:08:53.528 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/shm 2024-04-26T20:08:53.594 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/pts 2024-04-26T20:08:53.614 INFO:teuthology.orchestra.run.smithi094.stdout:/run 2024-04-26T20:08:53.624 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/cgroup 2024-04-26T20:08:53.636 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/pstore 2024-04-26T20:08:53.670 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/bpf 2024-04-26T20:08:53.680 INFO:teuthology.orchestra.run.smithi094.stdout:/ 2024-04-26T20:08:53.698 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/selinux 2024-04-26T20:08:53.771 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:08:53.852 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/hugepages 2024-04-26T20:08:53.863 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/mqueue 2024-04-26T20:08:53.873 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/debug 2024-04-26T20:08:53.885 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/tracing 2024-04-26T20:08:53.952 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/fuse/connections 2024-04-26T20:08:53.964 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/config 2024-04-26T20:08:53.978 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T20:08:53.991 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T20:08:54.000 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T20:08:54.041 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T20:08:54.089 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/1000 2024-04-26T20:08:54.100 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph 2024-04-26T20:08:54.112 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay 2024-04-26T20:08:54.124 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/418385bbe47ed7784b570b922aa87a7113b8db9a345aebb466844efbee55e0eb/merged 2024-04-26T20:08:54.135 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/3c4bbe9f866d0e96b3d0db8b3288b73a8fcabd18ea38a2a355887895541691a4/merged 2024-04-26T20:08:54.146 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/0 2024-04-26T20:08:54.223 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:08:54.270 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/949dab6365e3332593812ae709c31c8845515954e471282cab32f772637bb247/merged 2024-04-26T20:08:54.277 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/6f9c0eac9715a720f8c40fa431d23467c6ca9f2c1bb51be15b8541a822f31294/merged 2024-04-26T20:08:54.284 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/b93b38d85f3f58903119255d1ff9f2f4a0a5f5c5fdf77ab67d491cf27ab8d6b7/merged 2024-04-26T20:08:54.292 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/5910da20b2e6de27fbb4122e1124f1ce370f6f5c1f6480c0b0ca32f8aee50362/merged 2024-04-26T20:08:54.348 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/8aac3c8ab8dd4890bc525f11b8e553fabee687859d8b210bcb87e820b5525fd6/merged 2024-04-26T20:08:54.423 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/47531f1bd89f9bb2a8c026bb2a96d022375f2757d4f6f911394b9ae753eed69f/merged 2024-04-26T20:08:54.432 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/4a8dec9d7dbb1f22c889afa8f1ef5c31b31d8d77be70aa9285b1d6f9edd950e0/merged 2024-04-26T20:08:54.440 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/59bcd235986b201595e5e1754734839be11b6a5d514a59f2340c0f2cca8bc6da/merged 2024-04-26T20:08:54.497 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns 2024-04-26T20:08:54.572 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:08:54.586 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:08:54.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:54.616 DEBUG:teuthology.orchestra.run.smithi094:> ls /sys/fs/fuse/connections 2024-04-26T20:08:54.700 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-26T20:08:54.707 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-26T20:08:54.793 DEBUG:teuthology.orchestra.run.smithi094:> sudo modprobe fuse 2024-04-26T20:08:54.907 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:54 smithi094 ceph-mon[28050]: pgmap v343: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:54.918 DEBUG:teuthology.orchestra.run.smithi094:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T20:08:54.934 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:54 smithi204 ceph-mon[32618]: pgmap v343: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:54.973 INFO:teuthology.orchestra.run.smithi094.stdout:/proc 2024-04-26T20:08:54.994 INFO:teuthology.orchestra.run.smithi094.stdout:/sys 2024-04-26T20:08:55.005 INFO:teuthology.orchestra.run.smithi094.stdout:/dev 2024-04-26T20:08:55.019 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/security 2024-04-26T20:08:55.097 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/shm 2024-04-26T20:08:55.112 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/pts 2024-04-26T20:08:55.193 INFO:teuthology.orchestra.run.smithi094.stdout:/run 2024-04-26T20:08:55.204 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/cgroup 2024-04-26T20:08:55.216 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/pstore 2024-04-26T20:08:55.230 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/bpf 2024-04-26T20:08:55.243 INFO:teuthology.orchestra.run.smithi094.stdout:/ 2024-04-26T20:08:55.256 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/selinux 2024-04-26T20:08:55.267 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:08:55.347 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/hugepages 2024-04-26T20:08:55.421 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/mqueue 2024-04-26T20:08:55.430 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/debug 2024-04-26T20:08:55.440 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/tracing 2024-04-26T20:08:55.449 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/fuse/connections 2024-04-26T20:08:55.510 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/config 2024-04-26T20:08:55.558 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T20:08:55.571 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T20:08:55.635 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T20:08:55.648 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T20:08:55.697 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/1000 2024-04-26T20:08:55.709 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph 2024-04-26T20:08:55.722 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay 2024-04-26T20:08:55.734 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/418385bbe47ed7784b570b922aa87a7113b8db9a345aebb466844efbee55e0eb/merged 2024-04-26T20:08:55.745 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/3c4bbe9f866d0e96b3d0db8b3288b73a8fcabd18ea38a2a355887895541691a4/merged 2024-04-26T20:08:55.754 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/0 2024-04-26T20:08:55.766 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:08:55.826 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/949dab6365e3332593812ae709c31c8845515954e471282cab32f772637bb247/merged 2024-04-26T20:08:55.898 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/6f9c0eac9715a720f8c40fa431d23467c6ca9f2c1bb51be15b8541a822f31294/merged 2024-04-26T20:08:55.908 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/b93b38d85f3f58903119255d1ff9f2f4a0a5f5c5fdf77ab67d491cf27ab8d6b7/merged 2024-04-26T20:08:55.917 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/5910da20b2e6de27fbb4122e1124f1ce370f6f5c1f6480c0b0ca32f8aee50362/merged 2024-04-26T20:08:56.017 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/8aac3c8ab8dd4890bc525f11b8e553fabee687859d8b210bcb87e820b5525fd6/merged 2024-04-26T20:08:56.083 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/47531f1bd89f9bb2a8c026bb2a96d022375f2757d4f6f911394b9ae753eed69f/merged 2024-04-26T20:08:56.092 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/4a8dec9d7dbb1f22c889afa8f1ef5c31b31d8d77be70aa9285b1d6f9edd950e0/merged 2024-04-26T20:08:56.101 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/containers/storage/overlay/59bcd235986b201595e5e1754734839be11b6a5d514a59f2340c0f2cca8bc6da/merged 2024-04-26T20:08:56.109 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns 2024-04-26T20:08:56.118 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:08:56.169 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:08:56.177 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:54 smithi103 ceph-mon[32635]: pgmap v343: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:56.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:56.204 DEBUG:teuthology.orchestra.run.smithi094:> ls /sys/fs/fuse/connections 2024-04-26T20:08:56.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:2024-04-26T20:08:55.010+0000 7fcf2c920480 -1 init, newargv = 0x56444c28d870 newargc=15 2024-04-26T20:08:56.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:ceph-fuse[68972]: starting ceph client 2024-04-26T20:08:56.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:ceph-fuse[68972]: starting fuse 2024-04-26T20:08:56.393 INFO:teuthology.orchestra.run.smithi094.stdout:114 2024-04-26T20:08:56.478 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-04-26T20:08:56.489 DEBUG:teuthology.orchestra.run.smithi094:> sudo stdin-killer -- python3 -c ' 2024-04-26T20:08:56.496 DEBUG:teuthology.orchestra.run.smithi094:> import glob 2024-04-26T20:08:56.613 DEBUG:teuthology.orchestra.run.smithi094:> import re 2024-04-26T20:08:56.668 DEBUG:teuthology.orchestra.run.smithi094:> import os 2024-04-26T20:08:56.677 DEBUG:teuthology.orchestra.run.smithi094:> import subprocess 2024-04-26T20:08:56.686 DEBUG:teuthology.orchestra.run.smithi094:> 2024-04-26T20:08:56.694 DEBUG:teuthology.orchestra.run.smithi094:> def _find_admin_socket(client_name): 2024-04-26T20:08:56.702 DEBUG:teuthology.orchestra.run.smithi094:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-26T20:08:56.713 DEBUG:teuthology.orchestra.run.smithi094:> files = glob.glob(asok_path) 2024-04-26T20:08:56.721 DEBUG:teuthology.orchestra.run.smithi094:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-26T20:08:56.730 DEBUG:teuthology.orchestra.run.smithi094:> 2024-04-26T20:08:56.738 DEBUG:teuthology.orchestra.run.smithi094:> # Given a non-glob path, it better be there 2024-04-26T20:08:56.747 DEBUG:teuthology.orchestra.run.smithi094:> if "*" not in asok_path: 2024-04-26T20:08:56.755 DEBUG:teuthology.orchestra.run.smithi094:> assert(len(files) == 1) 2024-04-26T20:08:56.764 DEBUG:teuthology.orchestra.run.smithi094:> return files[0] 2024-04-26T20:08:56.853 DEBUG:teuthology.orchestra.run.smithi094:> 2024-04-26T20:08:56.931 DEBUG:teuthology.orchestra.run.smithi094:> for f in files: 2024-04-26T20:08:56.940 DEBUG:teuthology.orchestra.run.smithi094:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-26T20:08:56.949 DEBUG:teuthology.orchestra.run.smithi094:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-26T20:08:57.029 DEBUG:teuthology.orchestra.run.smithi094:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-26T20:08:57.078 DEBUG:teuthology.orchestra.run.smithi094:> contents = proc_f.read() 2024-04-26T20:08:57.086 DEBUG:teuthology.orchestra.run.smithi094:> if mountpoint in contents: 2024-04-26T20:08:57.155 DEBUG:teuthology.orchestra.run.smithi094:> return f 2024-04-26T20:08:57.166 DEBUG:teuthology.orchestra.run.smithi094:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-26T20:08:57.175 DEBUG:teuthology.orchestra.run.smithi094:> 2024-04-26T20:08:57.186 DEBUG:teuthology.orchestra.run.smithi094:> print(_find_admin_socket("client.admin")) 2024-04-26T20:08:57.195 DEBUG:teuthology.orchestra.run.smithi094:> ' 2024-04-26T20:08:57.204 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:56 smithi204 ceph-mon[32618]: pgmap v344: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:57.215 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:56 smithi103 ceph-mon[32635]: pgmap v344: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:57.224 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:56 smithi094 ceph-mon[28050]: pgmap v344: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:57.415 INFO:teuthology.orchestra.run.smithi094.stdout:/var/run/ceph/ceph-client.admin.68972.asok 2024-04-26T20:08:57.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:08:57.529 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.68972.asok 2024-04-26T20:08:57.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:57.619 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.68972.asok status 2024-04-26T20:08:57.827 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-26T20:08:57.849 INFO:teuthology.orchestra.run.smithi094.stdout: "metadata": { 2024-04-26T20:08:57.913 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_sha1": "bc612ffb9c1d5bc27536bc917a705262a36d4387", 2024-04-26T20:08:58.009 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T20:08:58.015 INFO:teuthology.orchestra.run.smithi094.stdout: "entity_id": "admin", 2024-04-26T20:08:58.023 INFO:teuthology.orchestra.run.smithi094.stdout: "hostname": "smithi094", 2024-04-26T20:08:58.030 INFO:teuthology.orchestra.run.smithi094.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-26T20:08:58.037 INFO:teuthology.orchestra.run.smithi094.stdout: "pid": "68972", 2024-04-26T20:08:58.043 INFO:teuthology.orchestra.run.smithi094.stdout: "root": "/" 2024-04-26T20:08:58.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-26T20:08:58.059 INFO:teuthology.orchestra.run.smithi094.stdout: "dentry_count": 0, 2024-04-26T20:08:58.065 INFO:teuthology.orchestra.run.smithi094.stdout: "dentry_pinned_count": 0, 2024-04-26T20:08:58.074 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24697, 2024-04-26T20:08:58.080 INFO:teuthology.orchestra.run.smithi094.stdout: "inst": { 2024-04-26T20:08:58.161 INFO:teuthology.orchestra.run.smithi094.stdout: "name": { 2024-04-26T20:08:58.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client", 2024-04-26T20:08:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 24697 2024-04-26T20:08:58.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-26T20:08:58.314 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": { 2024-04-26T20:08:58.371 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-04-26T20:08:58.425 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:0", 2024-04-26T20:08:58.434 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 2765551327 2024-04-26T20:08:58.442 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-04-26T20:08:58.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-26T20:08:58.457 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": { 2024-04-26T20:08:58.464 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-04-26T20:08:58.471 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "172.21.15.94:0", 2024-04-26T20:08:58.478 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 2765551327 2024-04-26T20:08:58.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-04-26T20:08:58.490 INFO:teuthology.orchestra.run.smithi094.stdout: "inst_str": "client.24697 172.21.15.94:0/2765551327", 2024-04-26T20:08:58.497 INFO:teuthology.orchestra.run.smithi094.stdout: "addr_str": "172.21.15.94:0/2765551327", 2024-04-26T20:08:58.584 INFO:teuthology.orchestra.run.smithi094.stdout: "inode_count": 1, 2024-04-26T20:08:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_epoch": 16, 2024-04-26T20:08:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_epoch": 92, 2024-04-26T20:08:58.651 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_epoch_barrier": 0, 2024-04-26T20:08:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "blocklisted": false, 2024-04-26T20:08:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs" 2024-04-26T20:08:58.821 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-26T20:08:58.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:58.841 DEBUG:teuthology.orchestra.run.smithi094:> stat --file-system '--printf=%T 2024-04-26T20:08:58.849 DEBUG:teuthology.orchestra.run.smithi094:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:08:58.915 INFO:teuthology.orchestra.run.smithi094.stdout:fuseblk 2024-04-26T20:08:58.924 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:08:58.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:08:58.942 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:08:58.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:08:58 smithi094 ceph-mon[28050]: pgmap v345: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:59.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:08:58 smithi103 ceph-mon[32635]: pgmap v345: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:59.043 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-26T20:08:59.142 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:08:58 smithi204 ceph-mon[32618]: pgmap v345: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:08:59.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T20:08:59.377 INFO:teuthology.orchestra.run.smithi094.stdout:# file: . 2024-04-26T20:08:59.385 INFO:teuthology.orchestra.run.smithi094.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-26T20:08:59.393 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:08:59.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:08:59.467 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:08:59.474 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:08:59.541 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:08:59.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.823+0000 7f6ca6400640 1 -- 172.21.15.94:0/715846676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0105b80 msgr2=0x7f6ca0106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:08:59.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.823+0000 7f6ca6400640 1 --2- 172.21.15.94:0/715846676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0105b80 0x7f6ca0106000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009860 tx=0x7f6c8802a340 comp rx=0 tx=0).stop 2024-04-26T20:08:59.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.824+0000 7f6ca6400640 1 -- 172.21.15.94:0/715846676 shutdown_connections 2024-04-26T20:08:59.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.824+0000 7f6ca6400640 1 --2- 172.21.15.94:0/715846676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ca0106540 0x7f6ca010cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.824+0000 7f6ca6400640 1 --2- 172.21.15.94:0/715846676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0105b80 0x7f6ca0106000 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:59.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.824+0000 7f6ca6400640 1 --2- 172.21.15.94:0/715846676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ca0104990 0x7f6ca0104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:08:59.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.824+0000 7f6ca6400640 1 -- 172.21.15.94:0/715846676 >> 172.21.15.94:0/715846676 conn(0x7f6ca00fff00 msgr2=0x7f6ca0102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:08:59.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.824+0000 7f6ca6400640 1 -- 172.21.15.94:0/715846676 shutdown_connections 2024-04-26T20:08:59.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.824+0000 7f6ca6400640 1 -- 172.21.15.94:0/715846676 wait complete. 2024-04-26T20:09:00.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.825+0000 7f6ca6400640 1 Processor -- start 2024-04-26T20:09:00.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.825+0000 7f6ca6400640 1 -- start start 2024-04-26T20:09:00.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.825+0000 7f6ca6400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0104990 0x7f6ca019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:00.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.825+0000 7f6ca6400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ca0105b80 0x7f6ca019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:00.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.825+0000 7f6ca6400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ca0106540 0x7f6ca01a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:00.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca6400640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01104a0 con 0x7f6ca0106540 2024-04-26T20:09:00.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca6400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca0110320 con 0x7f6ca0104990 2024-04-26T20:09:00.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca6400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca0110620 con 0x7f6ca0105b80 2024-04-26T20:09:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0104990 0x7f6ca019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:00.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0104990 0x7f6ca019abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:57396/0 (socket says 172.21.15.94:57396) 2024-04-26T20:09:00.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca5000640 1 -- 172.21.15.94:0/3758124125 learned_addr learned my addr 172.21.15.94:0/3758124125 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:00.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6c97e00640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ca0105b80 0x7f6ca019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:00.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca5a00640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ca0106540 0x7f6ca01a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:00.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca5000640 1 -- 172.21.15.94:0/3758124125 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ca0105b80 msgr2=0x7f6ca019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:00.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca5000640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ca0105b80 0x7f6ca019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:00.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca5000640 1 -- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ca0106540 msgr2=0x7f6ca01a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:00.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca5000640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ca0106540 0x7f6ca01a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:00.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.826+0000 7f6ca5000640 1 -- 172.21.15.94:0/3758124125 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca01a27a0 con 0x7f6ca0104990 2024-04-26T20:09:00.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.827+0000 7f6ca5000640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0104990 0x7f6ca019abd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e9c0 tx=0x7f6c9000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:00.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.827+0000 7f6c9e000640 1 -- 172.21.15.94:0/3758124125 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9000cd80 con 0x7f6ca0104990 2024-04-26T20:09:00.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.827+0000 7f6c9e000640 1 -- 172.21.15.94:0/3758124125 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c900035f0 con 0x7f6ca0104990 2024-04-26T20:09:00.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.827+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca01a2a90 con 0x7f6ca0104990 2024-04-26T20:09:00.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.827+0000 7f6c9e000640 1 -- 172.21.15.94:0/3758124125 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90010690 con 0x7f6ca0104990 2024-04-26T20:09:00.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.827+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca01a2fb0 con 0x7f6ca0104990 2024-04-26T20:09:00.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.828+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ca0069ac0 con 0x7f6ca0104990 2024-04-26T20:09:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.830+0000 7f6c9e000640 1 -- 172.21.15.94:0/3758124125 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6c900037a0 con 0x7f6ca0104990 2024-04-26T20:09:00.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.831+0000 7f6c9e000640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6c840a3bb0 0x7f6c840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:00.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.831+0000 7f6c9e000640 1 -- 172.21.15.94:0/3758124125 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f6c900d0a00 con 0x7f6ca0104990 2024-04-26T20:09:00.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.832+0000 7f6c97e00640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6c840a3bb0 0x7f6c840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:00.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.832+0000 7f6c97e00640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6c840a3bb0 0x7f6c840a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6c8802a930 tx=0x7f6c88030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:00.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:08:59.836+0000 7f6c9e000640 1 -- 172.21.15.94:0/3758124125 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6c9009d140 con 0x7f6ca0104990 2024-04-26T20:09:00.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.040+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ca0063530 con 0x7f6ca0104990 2024-04-26T20:09:00.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.042+0000 7f6c9e000640 1 -- 172.21.15.94:0/3758124125 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24165 (secure 0 0 0) 0x7f6c9009ca40 con 0x7f6ca0104990 2024-04-26T20:09:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:00.869 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":92,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:08:27.361832+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:09:00.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6c840a3bb0 msgr2=0x7f6c840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:00.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6c840a3bb0 0x7f6c840a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6c8802a930 tx=0x7f6c88030450 comp rx=0 tx=0).stop 2024-04-26T20:09:01.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0104990 msgr2=0x7f6ca019abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:01.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0104990 0x7f6ca019abd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e9c0 tx=0x7f6c9000ee80 comp rx=0 tx=0).stop 2024-04-26T20:09:01.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 shutdown_connections 2024-04-26T20:09:01.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6c840a3bb0 0x7f6c840a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ca0106540 0x7f6ca01a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:01.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ca0105b80 0x7f6ca019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:01.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 --2- 172.21.15.94:0/3758124125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ca0104990 0x7f6ca019abd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:01.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 >> 172.21.15.94:0/3758124125 conn(0x7f6ca00fff00 msgr2=0x7f6ca0108f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:01.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 shutdown_connections 2024-04-26T20:09:01.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:00.045+0000 7f6ca6400640 1 -- 172.21.15.94:0/3758124125 wait complete. 2024-04-26T20:09:01.211 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:08:13.904332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:08:15.670247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:08:23.682553+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:09:01.218 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:09:01.227 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:00 smithi204 ceph-mon[32618]: pgmap v346: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:01.256 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:00 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3758124125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:01.363 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:00 smithi103 ceph-mon[32635]: pgmap v346: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:01.372 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:00 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3758124125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:01.378 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:00 smithi094 ceph-mon[28050]: pgmap v346: 129 pgs: 129 active+clean; 579 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:01.444 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:00 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3758124125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:01.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.703+0000 7f2255800640 1 -- 172.21.15.94:0/425200416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f225010ee10 msgr2=0x7f2250111200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:01.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.703+0000 7f2255800640 1 --2- 172.21.15.94:0/425200416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f225010ee10 0x7f2250111200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2248009cd0 tx=0x7f224802a8e0 comp rx=0 tx=0).stop 2024-04-26T20:09:01.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.704+0000 7f2255800640 1 -- 172.21.15.94:0/425200416 shutdown_connections 2024-04-26T20:09:01.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.704+0000 7f2255800640 1 --2- 172.21.15.94:0/425200416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f225010ee10 0x7f2250111200 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:01.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.704+0000 7f2255800640 1 --2- 172.21.15.94:0/425200416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2250075390 0x7f2250075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:01.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.704+0000 7f2255800640 1 --2- 172.21.15.94:0/425200416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250076970 0x7f2250074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:01.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.704+0000 7f2255800640 1 -- 172.21.15.94:0/425200416 >> 172.21.15.94:0/425200416 conn(0x7f22500fddd0 msgr2=0x7f22501001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:01.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.704+0000 7f2255800640 1 -- 172.21.15.94:0/425200416 shutdown_connections 2024-04-26T20:09:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.704+0000 7f2255800640 1 -- 172.21.15.94:0/425200416 wait complete. 2024-04-26T20:09:01.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f2255800640 1 Processor -- start 2024-04-26T20:09:01.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f2255800640 1 -- start start 2024-04-26T20:09:01.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f2255800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2250075390 0x7f225019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:01.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f2255800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250076970 0x7f225019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:01.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f2255800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f225010ee10 0x7f22501a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:01.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f2255800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f22501149b0 con 0x7f2250075390 2024-04-26T20:09:01.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f2255800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2250114830 con 0x7f225010ee10 2024-04-26T20:09:01.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f2255800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2250114b30 con 0x7f2250076970 2024-04-26T20:09:02.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f224fe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2250075390 0x7f225019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:02.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f224fe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2250075390 0x7f225019f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52784/0 (socket says 172.21.15.94:52784) 2024-04-26T20:09:02.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f224fe00640 1 -- 172.21.15.94:0/133844486 learned_addr learned my addr 172.21.15.94:0/133844486 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:02.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.705+0000 7f2254e00640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f225010ee10 0x7f22501a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:02.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f2247400640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250076970 0x7f225019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:02.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f224fe00640 1 -- 172.21.15.94:0/133844486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250076970 msgr2=0x7f225019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:02.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f224fe00640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250076970 0x7f225019f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:02.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f224fe00640 1 -- 172.21.15.94:0/133844486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f225010ee10 msgr2=0x7f22501a65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:02.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f224fe00640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f225010ee10 0x7f22501a65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:02.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f224fe00640 1 -- 172.21.15.94:0/133844486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22501a6c20 con 0x7f2250075390 2024-04-26T20:09:02.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f2254e00640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f225010ee10 0x7f22501a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:09:02.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f224fe00640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2250075390 0x7f225019f000 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f224000e9f0 tx=0x7f224000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:02.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f224d600640 1 -- 172.21.15.94:0/133844486 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224000cd80 con 0x7f2250075390 2024-04-26T20:09:02.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f224d600640 1 -- 172.21.15.94:0/133844486 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22400054d0 con 0x7f2250075390 2024-04-26T20:09:02.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.706+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22501a6f10 con 0x7f2250075390 2024-04-26T20:09:02.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.707+0000 7f224d600640 1 -- 172.21.15.94:0/133844486 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2240010690 con 0x7f2250075390 2024-04-26T20:09:02.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.707+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22501a7320 con 0x7f2250075390 2024-04-26T20:09:02.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.708+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2250069af0 con 0x7f2250075390 2024-04-26T20:09:02.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.710+0000 7f224d600640 1 -- 172.21.15.94:0/133844486 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2240004e90 con 0x7f2250075390 2024-04-26T20:09:02.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.711+0000 7f224d600640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f22340a3bb0 0x7f22340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:02.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.712+0000 7f224d600640 1 -- 172.21.15.94:0/133844486 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f22400d0a90 con 0x7f2250075390 2024-04-26T20:09:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.712+0000 7f2247400640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f22340a3bb0 0x7f22340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:02.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.712+0000 7f2247400640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f22340a3bb0 0x7f22340a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f22380095f0 tx=0x7f2238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:02.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.716+0000 7f224d600640 1 -- 172.21.15.94:0/133844486 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2250069af0 con 0x7f2250075390 2024-04-26T20:09:02.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.921+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f22500635d0 con 0x7f2250075390 2024-04-26T20:09:02.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.921+0000 7f224d600640 1 -- 172.21.15.94:0/133844486 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f224009d1d0 con 0x7f2250075390 2024-04-26T20:09:02.616 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:09:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f22340a3bb0 msgr2=0x7f22340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:02.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f22340a3bb0 0x7f22340a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f22380095f0 tx=0x7f2238005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:02.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2250075390 msgr2=0x7f225019f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2250075390 0x7f225019f000 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f224000e9f0 tx=0x7f224000edf0 comp rx=0 tx=0).stop 2024-04-26T20:09:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 shutdown_connections 2024-04-26T20:09:02.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f22340a3bb0 0x7f22340a6070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:02.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f225010ee10 0x7f22501a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:02.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250076970 0x7f225019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:02.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 --2- 172.21.15.94:0/133844486 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2250075390 0x7f225019f000 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:02.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 >> 172.21.15.94:0/133844486 conn(0x7f22500fddd0 msgr2=0x7f22501020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:02.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 shutdown_connections 2024-04-26T20:09:02.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:01.924+0000 7f2255800640 1 -- 172.21.15.94:0/133844486 wait complete. 2024-04-26T20:09:02.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:09:03.002 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:02 smithi094 ceph-mon[28050]: pgmap v347: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:03.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:02 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/133844486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:09:03.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:02 smithi103 ceph-mon[32635]: pgmap v347: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:03.041 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:02 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/133844486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:09:03.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:02 smithi204 ceph-mon[32618]: pgmap v347: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:03.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:02 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/133844486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:09:03.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.233+0000 7f1673600640 1 -- 172.21.15.94:0/407011392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c076910 msgr2=0x7f166c074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:03.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.233+0000 7f1673600640 1 --2- 172.21.15.94:0/407011392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c076910 0x7f166c074d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f165c009860 tx=0x7f165c02a3d0 comp rx=0 tx=0).stop 2024-04-26T20:09:03.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.233+0000 7f1673600640 1 -- 172.21.15.94:0/407011392 shutdown_connections 2024-04-26T20:09:03.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.233+0000 7f1673600640 1 --2- 172.21.15.94:0/407011392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c10a810 0x7f166c10ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:03.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.234+0000 7f1673600640 1 --2- 172.21.15.94:0/407011392 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166c075330 0x7f166c075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:03.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.234+0000 7f1673600640 1 --2- 172.21.15.94:0/407011392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c076910 0x7f166c074d60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:03.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.234+0000 7f1673600640 1 -- 172.21.15.94:0/407011392 >> 172.21.15.94:0/407011392 conn(0x7f166c0fdf10 msgr2=0x7f166c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:03.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.234+0000 7f1673600640 1 -- 172.21.15.94:0/407011392 shutdown_connections 2024-04-26T20:09:03.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.234+0000 7f1673600640 1 -- 172.21.15.94:0/407011392 wait complete. 2024-04-26T20:09:03.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.235+0000 7f1673600640 1 Processor -- start 2024-04-26T20:09:03.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.235+0000 7f1673600640 1 -- start start 2024-04-26T20:09:03.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1673600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c075330 0x7f166c19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:03.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1673600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c076910 0x7f166c19b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:03.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1673600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166c10a810 0x7f166c1a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1673600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f166c1106d0 con 0x7f166c10a810 2024-04-26T20:09:03.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1673600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f166c110550 con 0x7f166c076910 2024-04-26T20:09:03.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1673600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f166c110850 con 0x7f166c075330 2024-04-26T20:09:03.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1671600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166c10a810 0x7f166c1a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:03.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1670c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c075330 0x7f166c19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:03.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f166be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c076910 0x7f166c19b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:03.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1670c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c075330 0x7f166c19ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:38798/0 (socket says 172.21.15.94:38798) 2024-04-26T20:09:03.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1670c00640 1 -- 172.21.15.94:0/453510676 learned_addr learned my addr 172.21.15.94:0/453510676 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:03.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.236+0000 7f1671600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166c10a810 0x7f166c1a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52808/0 (socket says 172.21.15.94:52808) 2024-04-26T20:09:03.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f166be00640 1 -- 172.21.15.94:0/453510676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c075330 msgr2=0x7f166c19ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:03.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f166be00640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c075330 0x7f166c19ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:03.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f166be00640 1 -- 172.21.15.94:0/453510676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166c10a810 msgr2=0x7f166c1a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:03.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f166be00640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166c10a810 0x7f166c1a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:04.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f166be00640 1 -- 172.21.15.94:0/453510676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c1a2a00 con 0x7f166c076910 2024-04-26T20:09:04.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f1670c00640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c075330 0x7f166c19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:09:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f166be00640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c076910 0x7f166c19b320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1660005fd0 tx=0x7f166000ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f1669600640 1 -- 172.21.15.94:0/453510676 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166000ccc0 con 0x7f166c076910 2024-04-26T20:09:04.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f1669600640 1 -- 172.21.15.94:0/453510676 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16600035f0 con 0x7f166c076910 2024-04-26T20:09:04.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166c1a2cf0 con 0x7f166c076910 2024-04-26T20:09:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f1669600640 1 -- 172.21.15.94:0/453510676 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166000ce60 con 0x7f166c076910 2024-04-26T20:09:04.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.237+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166c1a3230 con 0x7f166c076910 2024-04-26T20:09:04.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.239+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166c1095d0 con 0x7f166c076910 2024-04-26T20:09:04.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.241+0000 7f1669600640 1 -- 172.21.15.94:0/453510676 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1660003a80 con 0x7f166c076910 2024-04-26T20:09:04.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.242+0000 7f1669600640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16540a3c30 0x7f16540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:04.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.242+0000 7f1669600640 1 -- 172.21.15.94:0/453510676 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f16600d0990 con 0x7f166c076910 2024-04-26T20:09:04.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.242+0000 7f1670c00640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16540a3c30 0x7f16540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:04.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.243+0000 7f1670c00640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16540a3c30 0x7f16540a60f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f165c0095f0 tx=0x7f165c030980 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:04.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.245+0000 7f1669600640 1 -- 172.21.15.94:0/453510676 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f166009d0d0 con 0x7f166c076910 2024-04-26T20:09:04.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.447+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f166c02dd20 con 0x7f166c076910 2024-04-26T20:09:04.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.448+0000 7f1669600640 1 -- 172.21.15.94:0/453510676 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f166009c9d0 con 0x7f166c076910 2024-04-26T20:09:04.545 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:09:04.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.451+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16540a3c30 msgr2=0x7f16540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:04.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.451+0000 7f1673600640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16540a3c30 0x7f16540a60f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f165c0095f0 tx=0x7f165c030980 comp rx=0 tx=0).stop 2024-04-26T20:09:04.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.451+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c076910 msgr2=0x7f166c19b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:04.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.451+0000 7f1673600640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c076910 0x7f166c19b320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1660005fd0 tx=0x7f166000ed90 comp rx=0 tx=0).stop 2024-04-26T20:09:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.452+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 shutdown_connections 2024-04-26T20:09:04.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.452+0000 7f1673600640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16540a3c30 0x7f16540a60f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:04.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.452+0000 7f1673600640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166c10a810 0x7f166c1a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:04.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.452+0000 7f1673600640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c076910 0x7f166c19b320 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.452+0000 7f1673600640 1 --2- 172.21.15.94:0/453510676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c075330 0x7f166c19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:04.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.452+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 >> 172.21.15.94:0/453510676 conn(0x7f166c0fdf10 msgr2=0x7f166c1067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:04.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.452+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 shutdown_connections 2024-04-26T20:09:04.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:03.452+0000 7f1673600640 1 -- 172.21.15.94:0/453510676 wait complete. 2024-04-26T20:09:04.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:09:04.888 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:03 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/453510676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:09:04.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:03 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/453510676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:09:04.903 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:03 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/453510676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:09:04.971 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:04 smithi094 ceph-mon[28050]: pgmap v348: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:05.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:04 smithi103 ceph-mon[32635]: pgmap v348: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:05.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.109+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3560924954 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7f010a850 msgr2=0x7ff7f010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:05.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.109+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3560924954 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7f010a850 0x7ff7f010cce0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009900 tx=0x7ff7e802a480 comp rx=0 tx=0).stop 2024-04-26T20:09:05.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.109+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3560924954 shutdown_connections 2024-04-26T20:09:05.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.109+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3560924954 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7f010a850 0x7ff7f010cce0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:05.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.109+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3560924954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7f0075310 0x7ff7f0075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:05.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.109+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3560924954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f00768f0 0x7ff7f0074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:05.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.109+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3560924954 >> 172.21.15.94:0/3560924954 conn(0x7ff7f00fdef0 msgr2=0x7ff7f0100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:05.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.109+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3560924954 shutdown_connections 2024-04-26T20:09:05.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.109+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3560924954 wait complete. 2024-04-26T20:09:05.522 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:04 smithi204 ceph-mon[32618]: pgmap v348: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:05.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.110+0000 7ff7f4a00640 1 Processor -- start 2024-04-26T20:09:05.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.110+0000 7ff7f4a00640 1 -- start start 2024-04-26T20:09:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.110+0000 7ff7f4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f0075310 0x7ff7f019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:05.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.110+0000 7ff7f4a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7f00768f0 0x7ff7f019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:05.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7f4a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7f010a850 0x7ff7f01a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:05.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7f4a00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f01106e0 con 0x7ff7f010a850 2024-04-26T20:09:05.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7f4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0110560 con 0x7ff7f0075310 2024-04-26T20:09:05.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7f4a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0110860 con 0x7ff7f00768f0 2024-04-26T20:09:05.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ede00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f0075310 0x7ff7f019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:05.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ee800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7f010a850 0x7ff7f01a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ede00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f0075310 0x7ff7f019ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:59092/0 (socket says 172.21.15.94:59092) 2024-04-26T20:09:05.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ede00640 1 -- 172.21.15.94:0/3963017959 learned_addr learned my addr 172.21.15.94:0/3963017959 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:05.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ed400640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7f00768f0 0x7ff7f019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:05.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ede00640 1 -- 172.21.15.94:0/3963017959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7f00768f0 msgr2=0x7ff7f019b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:05.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ede00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7f00768f0 0x7ff7f019b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:05.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ede00640 1 -- 172.21.15.94:0/3963017959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7f010a850 msgr2=0x7ff7f01a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:05.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ede00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7f010a850 0x7ff7f01a2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:06.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.111+0000 7ff7ede00640 1 -- 172.21.15.94:0/3963017959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f01a2970 con 0x7ff7f0075310 2024-04-26T20:09:06.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.112+0000 7ff7ede00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f0075310 0x7ff7f019ad50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000e9f0 tx=0x7ff7e000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.112+0000 7ff7e6a00640 1 -- 172.21.15.94:0/3963017959 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e000cd80 con 0x7ff7f0075310 2024-04-26T20:09:06.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.112+0000 7ff7e6a00640 1 -- 172.21.15.94:0/3963017959 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7e00108e0 con 0x7ff7f0075310 2024-04-26T20:09:06.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.112+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f01a2c60 con 0x7ff7f0075310 2024-04-26T20:09:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.112+0000 7ff7e6a00640 1 -- 172.21.15.94:0/3963017959 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0018980 con 0x7ff7f0075310 2024-04-26T20:09:06.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.112+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f01a31b0 con 0x7ff7f0075310 2024-04-26T20:09:06.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.113+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f0109610 con 0x7ff7f0075310 2024-04-26T20:09:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.115+0000 7ff7e6a00640 1 -- 172.21.15.94:0/3963017959 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff7e0010430 con 0x7ff7f0075310 2024-04-26T20:09:06.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.116+0000 7ff7e6a00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff7d40a3c30 0x7ff7d40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:06.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.117+0000 7ff7e6a00640 1 -- 172.21.15.94:0/3963017959 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7ff7e00d0960 con 0x7ff7f0075310 2024-04-26T20:09:06.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.117+0000 7ff7ed400640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff7d40a3c30 0x7ff7d40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:06.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.117+0000 7ff7ed400640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff7d40a3c30 0x7ff7d40a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff7f019c260 tx=0x7ff7dc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:06.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.121+0000 7ff7e6a00640 1 -- 172.21.15.94:0/3963017959 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff7e009ca50 con 0x7ff7f0075310 2024-04-26T20:09:06.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.323+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff7f00635d0 con 0x7ff7f0075310 2024-04-26T20:09:06.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.323+0000 7ff7e6a00640 1 -- 172.21.15.94:0/3963017959 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7ff7e00a0d50 con 0x7ff7f0075310 2024-04-26T20:09:06.535 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-26T20:09:06.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff7d40a3c30 msgr2=0x7ff7d40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:06.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff7d40a3c30 0x7ff7d40a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff7f019c260 tx=0x7ff7dc009500 comp rx=0 tx=0).stop 2024-04-26T20:09:06.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f0075310 msgr2=0x7ff7f019ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:06.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f0075310 0x7ff7f019ad50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000e9f0 tx=0x7ff7e000eeb0 comp rx=0 tx=0).stop 2024-04-26T20:09:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 shutdown_connections 2024-04-26T20:09:06.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff7d40a3c30 0x7ff7d40a60f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:06.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7f010a850 0x7ff7f01a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:06.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7f00768f0 0x7ff7f019b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:06.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 --2- 172.21.15.94:0/3963017959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7f0075310 0x7ff7f019ad50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:06.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 >> 172.21.15.94:0/3963017959 conn(0x7ff7f00fdef0 msgr2=0x7ff7f01067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:06.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 shutdown_connections 2024-04-26T20:09:06.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:05.326+0000 7ff7f4a00640 1 -- 172.21.15.94:0/3963017959 wait complete. 2024-04-26T20:09:06.799 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:09:06.858 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:05 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3963017959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:09:06.870 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:05 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3963017959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:09:06.922 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:05 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3963017959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:09:06.986 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:06 smithi094 ceph-mon[28050]: pgmap v349: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:07.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:06 smithi103 ceph-mon[32635]: pgmap v349: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:07.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.080+0000 7fdc05000640 1 -- 172.21.15.94:0/3436882672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 msgr2=0x7fdc0010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:07.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.080+0000 7fdc05000640 1 --2- 172.21.15.94:0/3436882672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 0x7fdc0010a600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009900 tx=0x7fdbe802a400 comp rx=0 tx=0).stop 2024-04-26T20:09:07.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.080+0000 7fdc05000640 1 -- 172.21.15.94:0/3436882672 shutdown_connections 2024-04-26T20:09:07.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.080+0000 7fdc05000640 1 --2- 172.21.15.94:0/3436882672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc0010ab40 0x7fdc001113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:07.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.080+0000 7fdc05000640 1 --2- 172.21.15.94:0/3436882672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 0x7fdc0010a600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:07.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.080+0000 7fdc05000640 1 --2- 172.21.15.94:0/3436882672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc00108f90 0x7fdc001093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:07.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.080+0000 7fdc05000640 1 -- 172.21.15.94:0/3436882672 >> 172.21.15.94:0/3436882672 conn(0x7fdc00077f30 msgr2=0x7fdc0007a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.080+0000 7fdc05000640 1 -- 172.21.15.94:0/3436882672 shutdown_connections 2024-04-26T20:09:07.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.081+0000 7fdc05000640 1 -- 172.21.15.94:0/3436882672 wait complete. 2024-04-26T20:09:07.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:06 smithi204 ceph-mon[32618]: pgmap v349: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:07.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.081+0000 7fdc05000640 1 Processor -- start 2024-04-26T20:09:07.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdc05000640 1 -- start start 2024-04-26T20:09:07.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdc05000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc00108f90 0x7fdc0019f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:07.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdc05000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 0x7fdc0019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:07.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdc05000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0010ab40 0x7fdc001a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:07.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdc05000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00114b20 con 0x7fdc0010a180 2024-04-26T20:09:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdc05000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001149a0 con 0x7fdc00108f90 2024-04-26T20:09:07.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdc05000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00114ca0 con 0x7fdc0010ab40 2024-04-26T20:09:07.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdbfd400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 0x7fdc0019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:07.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdbfd400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 0x7fdc0019f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52846/0 (socket says 172.21.15.94:52846) 2024-04-26T20:09:07.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdbfd400640 1 -- 172.21.15.94:0/979833952 learned_addr learned my addr 172.21.15.94:0/979833952 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:07.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdbfde00640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc00108f90 0x7fdc0019f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:07.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.082+0000 7fdbfe800640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0010ab40 0x7fdc001a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:07.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdbfd400640 1 -- 172.21.15.94:0/979833952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0010ab40 msgr2=0x7fdc001a67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:07.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdbfd400640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0010ab40 0x7fdc001a67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:07.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdbfd400640 1 -- 172.21.15.94:0/979833952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc00108f90 msgr2=0x7fdc0019f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:07.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdbfd400640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc00108f90 0x7fdc0019f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:07.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdbfd400640 1 -- 172.21.15.94:0/979833952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc001a6e20 con 0x7fdc0010a180 2024-04-26T20:09:07.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdbfd400640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 0x7fdc0019f740 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8005ab0 tx=0x7fdbe8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:07.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdbf6a00640 1 -- 172.21.15.94:0/979833952 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe803f070 con 0x7fdc0010a180 2024-04-26T20:09:07.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc001a70b0 con 0x7fdc0010a180 2024-04-26T20:09:07.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdbf6a00640 1 -- 172.21.15.94:0/979833952 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbe8031c50 con 0x7fdc0010a180 2024-04-26T20:09:08.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdbf6a00640 1 -- 172.21.15.94:0/979833952 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8035030 con 0x7fdc0010a180 2024-04-26T20:09:08.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.083+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc001a75a0 con 0x7fdc0010a180 2024-04-26T20:09:08.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.084+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc001102d0 con 0x7fdc0010a180 2024-04-26T20:09:08.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.086+0000 7fdbf6a00640 1 -- 172.21.15.94:0/979833952 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdbe8042070 con 0x7fdc0010a180 2024-04-26T20:09:08.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.088+0000 7fdbf6a00640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdbe40a3bb0 0x7fdbe40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:08.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.088+0000 7fdbf6a00640 1 -- 172.21.15.94:0/979833952 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fdbe80f0e70 con 0x7fdc0010a180 2024-04-26T20:09:08.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.088+0000 7fdbfde00640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdbe40a3bb0 0x7fdbe40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:08.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.089+0000 7fdbfde00640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdbe40a3bb0 0x7fdbe40a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0005fd0 tx=0x7fdbf000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:08.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.092+0000 7fdbf6a00640 1 -- 172.21.15.94:0/979833952 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdbe80bcf60 con 0x7fdc0010a180 2024-04-26T20:09:08.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.295+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fdc00063530 con 0x7fdc0010a180 2024-04-26T20:09:08.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.296+0000 7fdbf6a00640 1 -- 172.21.15.94:0/979833952 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7fdbe80c1260 con 0x7fdc0010a180 2024-04-26T20:09:08.184 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:09:08.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdbe40a3bb0 msgr2=0x7fdbe40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:08.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdbe40a3bb0 0x7fdbe40a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0005fd0 tx=0x7fdbf000b390 comp rx=0 tx=0).stop 2024-04-26T20:09:08.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 msgr2=0x7fdc0019f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:08.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 0x7fdc0019f740 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8005ab0 tx=0x7fdbe8003b90 comp rx=0 tx=0).stop 2024-04-26T20:09:08.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 shutdown_connections 2024-04-26T20:09:08.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdbe40a3bb0 0x7fdbe40a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:08.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0010ab40 0x7fdc001a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:08.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc0010a180 0x7fdc0019f740 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:08.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 --2- 172.21.15.94:0/979833952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc00108f90 0x7fdc0019f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:08.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 >> 172.21.15.94:0/979833952 conn(0x7fdc00077f30 msgr2=0x7fdc0010d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:08.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 shutdown_connections 2024-04-26T20:09:08.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:07.299+0000 7fdc05000640 1 -- 172.21.15.94:0/979833952 wait complete. 2024-04-26T20:09:08.574 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T20:09:08.581 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:07 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/979833952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:09:08.587 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:07 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/979833952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:09:08.593 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:07 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/979833952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:09:08.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.770+0000 7f04cf600640 1 -- 172.21.15.94:0/2756012027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c8106760 msgr2=0x7f04c810cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:08.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.770+0000 7f04cf600640 1 --2- 172.21.15.94:0/2756012027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c8106760 0x7f04c810cff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f04c0009be0 tx=0x7f04c002a6c0 comp rx=0 tx=0).stop 2024-04-26T20:09:08.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.771+0000 7f04cf600640 1 -- 172.21.15.94:0/2756012027 shutdown_connections 2024-04-26T20:09:08.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.771+0000 7f04cf600640 1 --2- 172.21.15.94:0/2756012027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c8106760 0x7f04c810cff0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:08.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.771+0000 7f04cf600640 1 --2- 172.21.15.94:0/2756012027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8105da0 0x7f04c8106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:08.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.771+0000 7f04cf600640 1 --2- 172.21.15.94:0/2756012027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04c8104bb0 0x7f04c8104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:08.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.771+0000 7f04cf600640 1 -- 172.21.15.94:0/2756012027 >> 172.21.15.94:0/2756012027 conn(0x7f04c8100120 msgr2=0x7f04c8102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:09.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.771+0000 7f04cf600640 1 -- 172.21.15.94:0/2756012027 shutdown_connections 2024-04-26T20:09:09.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.771+0000 7f04cf600640 1 -- 172.21.15.94:0/2756012027 wait complete. 2024-04-26T20:09:09.068 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:08 smithi103 ceph-mon[32635]: pgmap v350: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:09.078 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:09:09.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04cf600640 1 Processor -- start 2024-04-26T20:09:09.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04cf600640 1 -- start start 2024-04-26T20:09:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04cf600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8104bb0 0x7f04c819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:09.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04cf600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04c8105da0 0x7f04c819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:09.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04cf600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c8106760 0x7f04c81a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:09.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04cf600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f04c81106c0 con 0x7f04c8104bb0 2024-04-26T20:09:09.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04cf600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04c8110540 con 0x7f04c8106760 2024-04-26T20:09:09.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04cf600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f04c8110840 con 0x7f04c8105da0 2024-04-26T20:09:09.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04c7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04c8105da0 0x7f04c819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:09.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04ccc00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8104bb0 0x7f04c819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:09.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04c7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04c8105da0 0x7f04c819b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:38854/0 (socket says 172.21.15.94:38854) 2024-04-26T20:09:09.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04c7e00640 1 -- 172.21.15.94:0/2937555010 learned_addr learned my addr 172.21.15.94:0/2937555010 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:09.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.772+0000 7f04cd600640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c8106760 0x7f04c81a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04c7e00640 1 -- 172.21.15.94:0/2937555010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c8106760 msgr2=0x7f04c81a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04c7e00640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c8106760 0x7f04c81a2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04c7e00640 1 -- 172.21.15.94:0/2937555010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8104bb0 msgr2=0x7f04c819ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:09.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04c7e00640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8104bb0 0x7f04c819ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:09.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04c7e00640 1 -- 172.21.15.94:0/2937555010 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c81a2970 con 0x7f04c8105da0 2024-04-26T20:09:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04c7e00640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04c8105da0 0x7f04c819b290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f04b400e9c0 tx=0x7f04b400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:09.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04c5600640 1 -- 172.21.15.94:0/2937555010 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b400ccc0 con 0x7f04c8105da0 2024-04-26T20:09:09.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04c5600640 1 -- 172.21.15.94:0/2937555010 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04b40054d0 con 0x7f04c8105da0 2024-04-26T20:09:09.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04c5600640 1 -- 172.21.15.94:0/2937555010 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4010430 con 0x7f04c8105da0 2024-04-26T20:09:09.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.773+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c81a2c60 con 0x7f04c8105da0 2024-04-26T20:09:09.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.774+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c81a31b0 con 0x7f04c8105da0 2024-04-26T20:09:09.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.775+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c810bfa0 con 0x7f04c8105da0 2024-04-26T20:09:09.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.777+0000 7f04c5600640 1 -- 172.21.15.94:0/2937555010 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f04b4004f80 con 0x7f04c8105da0 2024-04-26T20:09:09.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.778+0000 7f04c5600640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04b00a3c30 0x7f04b00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:09.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.778+0000 7f04c5600640 1 -- 172.21.15.94:0/2937555010 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f04b40d09e0 con 0x7f04c8105da0 2024-04-26T20:09:09.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.778+0000 7f04ccc00640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04b00a3c30 0x7f04b00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:10.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.779+0000 7f04ccc00640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04b00a3c30 0x7f04b00a60f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f04bc0095f0 tx=0x7f04bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:10.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.782+0000 7f04c5600640 1 -- 172.21.15.94:0/2937555010 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f04b409d120 con 0x7f04c8105da0 2024-04-26T20:09:10.105 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:08 smithi094 ceph-mon[28050]: pgmap v350: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:10.113 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:09:10.122 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:09 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2937555010' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:09:10.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:08 smithi204 ceph-mon[32618]: pgmap v350: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:10.138 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:09:10.146 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:09 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2937555010' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:09:10.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.998+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f04c802dad0 con 0x7f04c8105da0 2024-04-26T20:09:10.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:08.999+0000 7f04c5600640 1 -- 172.21.15.94:0/2937555010 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v92) ==== 81+0+591 (secure 0 0 0) 0x7f04b409ca20 con 0x7f04c8105da0 2024-04-26T20:09:10.166 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3877259400 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:10.172 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3361956945 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:10.319 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3651438916 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:10.405 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3005317891 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:10.417 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1688404742 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:10.480 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/2041870658 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:10.538 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/442392134 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:10.570 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1853030572 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:10.577 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3005317891 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:10.583 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3651438916 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:10.592 INFO:teuthology.orchestra.run.smithi094.stderr:listed 10 entries 2024-04-26T20:09:10.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04b00a3c30 msgr2=0x7f04b00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:10.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04b00a3c30 0x7f04b00a60f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f04bc0095f0 tx=0x7f04bc005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:10.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04c8105da0 msgr2=0x7f04c819b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:10.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04c8105da0 0x7f04c819b290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f04b400e9c0 tx=0x7f04b400ee80 comp rx=0 tx=0).stop 2024-04-26T20:09:10.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 shutdown_connections 2024-04-26T20:09:10.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04b00a3c30 0x7f04b00a60f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04c8106760 0x7f04c81a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:10.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04c8105da0 0x7f04c819b290 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:10.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 --2- 172.21.15.94:0/2937555010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04c8104bb0 0x7f04c819ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:10.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.002+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 >> 172.21.15.94:0/2937555010 conn(0x7f04c8100120 msgr2=0x7f04c8109170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.003+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 shutdown_connections 2024-04-26T20:09:10.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:09.003+0000 7f04cf600640 1 -- 172.21.15.94:0/2937555010 wait complete. 2024-04-26T20:09:10.995 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:09:11.052 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:09:11.062 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:09 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2937555010' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:09:11.071 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:10 smithi094 ceph-mon[28050]: pgmap v351: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:11.081 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.089 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.098 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.106 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:10 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:10 smithi103 ceph-mon[32635]: pgmap v351: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:11.202 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.210 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.217 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:10 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:10 smithi204 ceph-mon[32618]: pgmap v351: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:11.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.392 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.495 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.542 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:10 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:11.610 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:09:11.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.851+0000 7ffa8d600640 1 -- 172.21.15.94:0/2473107905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 msgr2=0x7ffa88103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:11.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.851+0000 7ffa8d600640 1 --2- 172.21.15.94:0/2473107905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 0x7ffa88103de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009900 tx=0x7ffa7402a400 comp rx=0 tx=0).stop 2024-04-26T20:09:11.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.851+0000 7ffa8d600640 1 -- 172.21.15.94:0/2473107905 shutdown_connections 2024-04-26T20:09:11.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.851+0000 7ffa8d600640 1 --2- 172.21.15.94:0/2473107905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa88104320 0x7ffa8810abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:11.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.851+0000 7ffa8d600640 1 --2- 172.21.15.94:0/2473107905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 0x7ffa88103de0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:11.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.851+0000 7ffa8d600640 1 --2- 172.21.15.94:0/2473107905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa88102770 0x7ffa88102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:11.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.851+0000 7ffa8d600640 1 -- 172.21.15.94:0/2473107905 >> 172.21.15.94:0/2473107905 conn(0x7ffa880fdcc0 msgr2=0x7ffa88100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.851+0000 7ffa8d600640 1 -- 172.21.15.94:0/2473107905 shutdown_connections 2024-04-26T20:09:11.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.851+0000 7ffa8d600640 1 -- 172.21.15.94:0/2473107905 wait complete. 2024-04-26T20:09:11.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.853+0000 7ffa8d600640 1 Processor -- start 2024-04-26T20:09:12.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.853+0000 7ffa8d600640 1 -- start start 2024-04-26T20:09:12.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa8d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa88102770 0x7ffa880766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa8d600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 0x7ffa88074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa8d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa88104320 0x7ffa88075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa8d600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8810e2f0 con 0x7ffa88103960 2024-04-26T20:09:12.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa8d600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8810e170 con 0x7ffa88102770 2024-04-26T20:09:12.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa8d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8810e470 con 0x7ffa88104320 2024-04-26T20:09:12.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa87400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 0x7ffa88074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa87400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 0x7ffa88074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34516/0 (socket says 172.21.15.94:34516) 2024-04-26T20:09:12.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa87400640 1 -- 172.21.15.94:0/148069182 learned_addr learned my addr 172.21.15.94:0/148069182 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:12.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa87e00640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa88102770 0x7ffa880766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa8cc00640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa88104320 0x7ffa88075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa87400640 1 -- 172.21.15.94:0/148069182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa88104320 msgr2=0x7ffa88075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa87400640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa88104320 0x7ffa88075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa87400640 1 -- 172.21.15.94:0/148069182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa88102770 msgr2=0x7ffa880766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa87400640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa88102770 0x7ffa880766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa87400640 1 -- 172.21.15.94:0/148069182 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa88075af0 con 0x7ffa88103960 2024-04-26T20:09:12.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.854+0000 7ffa8cc00640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa88104320 0x7ffa88075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:09:12.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.855+0000 7ffa87400640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 0x7ffa88074d20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffa74005ab0 tx=0x7ffa74003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:12.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.855+0000 7ffa84c00640 1 -- 172.21.15.94:0/148069182 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7403f070 con 0x7ffa88103960 2024-04-26T20:09:12.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.855+0000 7ffa84c00640 1 -- 172.21.15.94:0/148069182 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa74031c50 con 0x7ffa88103960 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.855+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa881a2a20 con 0x7ffa88103960 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.855+0000 7ffa84c00640 1 -- 172.21.15.94:0/148069182 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74035030 con 0x7ffa88103960 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.855+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa881a2f70 con 0x7ffa88103960 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.857+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa88109b10 con 0x7ffa88103960 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.858+0000 7ffa84c00640 1 -- 172.21.15.94:0/148069182 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffa74042070 con 0x7ffa88103960 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.859+0000 7ffa84c00640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffa6c0a3bb0 0x7ffa6c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.860+0000 7ffa84c00640 1 -- 172.21.15.94:0/148069182 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7ffa740f0f40 con 0x7ffa88103960 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.860+0000 7ffa87e00640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffa6c0a3bb0 0x7ffa6c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.860+0000 7ffa87e00640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffa6c0a3bb0 0x7ffa6c0a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffa78005fd0 tx=0x7ffa7800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:12.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:11.864+0000 7ffa84c00640 1 -- 172.21.15.94:0/148069182 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffa740bd680 con 0x7ffa88103960 2024-04-26T20:09:12.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.136+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa88063530 con 0x7ffa88103960 2024-04-26T20:09:12.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.141+0000 7ffa84c00640 1 -- 172.21.15.94:0/148069182 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24165 (secure 0 0 0) 0x7ffa740bcf80 con 0x7ffa88103960 2024-04-26T20:09:12.325 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:12.325 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":92,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:08:27.361832+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:09:12.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.143+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffa6c0a3bb0 msgr2=0x7ffa6c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.143+0000 7ffa8d600640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffa6c0a3bb0 0x7ffa6c0a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffa78005fd0 tx=0x7ffa7800b390 comp rx=0 tx=0).stop 2024-04-26T20:09:12.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.143+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 msgr2=0x7ffa88074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.143+0000 7ffa8d600640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 0x7ffa88074d20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffa74005ab0 tx=0x7ffa74003b90 comp rx=0 tx=0).stop 2024-04-26T20:09:12.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.144+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 shutdown_connections 2024-04-26T20:09:12.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.144+0000 7ffa8d600640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ffa6c0a3bb0 0x7ffa6c0a6070 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.144+0000 7ffa8d600640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa88104320 0x7ffa88075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.144+0000 7ffa8d600640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa88103960 0x7ffa88074d20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.144+0000 7ffa8d600640 1 --2- 172.21.15.94:0/148069182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa88102770 0x7ffa880766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.144+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 >> 172.21.15.94:0/148069182 conn(0x7ffa880fdcc0 msgr2=0x7ffa88106ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:12.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.144+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 shutdown_connections 2024-04-26T20:09:12.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.144+0000 7ffa8d600640 1 -- 172.21.15.94:0/148069182 wait complete. 2024-04-26T20:09:12.328 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:08:13.904332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:08:15.670247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:08:23.682553+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:09:12.328 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:09:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2024-04-26T20:09:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-26T20:09:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-26T20:09:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-04-26T20:09:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-26T20:09:12.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-04-26T20:09:12.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-04-26T20:09:12.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-04-26T20:09:12.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-26T20:09:12.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-26T20:09:12.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: pgmap v352: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:12.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:12 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/148069182' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:12.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.518+0000 7f1bdb200640 1 -- 172.21.15.94:0/2345686131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4103b70 msgr2=0x7f1bd4103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.518+0000 7f1bdb200640 1 --2- 172.21.15.94:0/2345686131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4103b70 0x7f1bd4103ff0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009900 tx=0x7f1bc002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.518+0000 7f1bdb200640 1 -- 172.21.15.94:0/2345686131 shutdown_connections 2024-04-26T20:09:12.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.518+0000 7f1bdb200640 1 --2- 172.21.15.94:0/2345686131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd4104530 0x7f1bd410adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.518+0000 7f1bdb200640 1 --2- 172.21.15.94:0/2345686131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4103b70 0x7f1bd4103ff0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.518+0000 7f1bdb200640 1 --2- 172.21.15.94:0/2345686131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd4102980 0x7f1bd4102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.518+0000 7f1bdb200640 1 -- 172.21.15.94:0/2345686131 >> 172.21.15.94:0/2345686131 conn(0x7f1bd40fdf10 msgr2=0x7f1bd4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:12.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.518+0000 7f1bdb200640 1 -- 172.21.15.94:0/2345686131 shutdown_connections 2024-04-26T20:09:12.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.518+0000 7f1bdb200640 1 -- 172.21.15.94:0/2345686131 wait complete. 2024-04-26T20:09:12.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.519+0000 7f1bdb200640 1 Processor -- start 2024-04-26T20:09:12.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.519+0000 7f1bdb200640 1 -- start start 2024-04-26T20:09:12.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bdb200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4102980 0x7f1bd4198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bdb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd4103b70 0x7f1bd4199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bdb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd4104530 0x7f1bd41a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bdb200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd410e490 con 0x7f1bd4102980 2024-04-26T20:09:12.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bdb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd410e310 con 0x7f1bd4104530 2024-04-26T20:09:12.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bdb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd410e610 con 0x7f1bd4103b70 2024-04-26T20:09:12.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bd3e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4102980 0x7f1bd4198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bd3e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4102980 0x7f1bd4198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34526/0 (socket says 172.21.15.94:34526) 2024-04-26T20:09:12.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bd3e00640 1 -- 172.21.15.94:0/837880147 learned_addr learned my addr 172.21.15.94:0/837880147 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:12.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bd3400640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd4103b70 0x7f1bd4199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bd9200640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd4104530 0x7f1bd41a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bd3e00640 1 -- 172.21.15.94:0/837880147 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd4103b70 msgr2=0x7f1bd4199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.520+0000 7f1bd3e00640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd4103b70 0x7f1bd4199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.521+0000 7f1bd3e00640 1 -- 172.21.15.94:0/837880147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd4104530 msgr2=0x7f1bd41a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.521+0000 7f1bd3e00640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd4104530 0x7f1bd41a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.521+0000 7f1bd3e00640 1 -- 172.21.15.94:0/837880147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd41a07f0 con 0x7f1bd4102980 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.521+0000 7f1bd3e00640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4102980 0x7f1bd4198bd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400d950 tx=0x7f1bc400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.521+0000 7f1bd0c00640 1 -- 172.21.15.94:0/837880147 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4017020 con 0x7f1bd4102980 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.521+0000 7f1bd0c00640 1 -- 172.21.15.94:0/837880147 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bc4009c30 con 0x7f1bd4102980 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.521+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd41a0ae0 con 0x7f1bd4102980 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.521+0000 7f1bd0c00640 1 -- 172.21.15.94:0/837880147 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc400f690 con 0x7f1bd4102980 2024-04-26T20:09:12.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.521+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd41a1000 con 0x7f1bd4102980 2024-04-26T20:09:12.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.522+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd4109dc0 con 0x7f1bd4102980 2024-04-26T20:09:12.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.524+0000 7f1bd0c00640 1 -- 172.21.15.94:0/837880147 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1bc401a020 con 0x7f1bd4102980 2024-04-26T20:09:12.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.526+0000 7f1bd0c00640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1bb80a3bb0 0x7f1bb80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.526+0000 7f1bd0c00640 1 -- 172.21.15.94:0/837880147 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1bc40d0600 con 0x7f1bd4102980 2024-04-26T20:09:12.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.526+0000 7f1bd3400640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1bb80a3bb0 0x7f1bb80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.527+0000 7f1bd3400640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1bb80a3bb0 0x7f1bb80a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0000c00 tx=0x7f1bc00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:12.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.530+0000 7f1bd0c00640 1 -- 172.21.15.94:0/837880147 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1bc4021020 con 0x7f1bd4102980 2024-04-26T20:09:12.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:12.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:12.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2024-04-26T20:09:12.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-26T20:09:12.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-26T20:09:12.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-04-26T20:09:12.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-26T20:09:12.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-04-26T20:09:12.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-04-26T20:09:12.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-04-26T20:09:12.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-26T20:09:12.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-26T20:09:12.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: pgmap v352: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:12.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:12 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/148069182' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-04-26T20:09:12.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-26T20:09:12.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-26T20:09:12.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: pgmap v352: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:09:12.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:12 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/148069182' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:12.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.732+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1bd40635d0 con 0x7f1bd4102980 2024-04-26T20:09:12.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.733+0000 7f1bd0c00640 1 -- 172.21.15.94:0/837880147 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f1bc4009de0 con 0x7f1bd4102980 2024-04-26T20:09:12.734 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:09:12.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1bb80a3bb0 msgr2=0x7f1bb80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1bb80a3bb0 0x7f1bb80a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0000c00 tx=0x7f1bc00314a0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4102980 msgr2=0x7f1bd4198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4102980 0x7f1bd4198bd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400d950 tx=0x7f1bc400de10 comp rx=0 tx=0).stop 2024-04-26T20:09:12.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 shutdown_connections 2024-04-26T20:09:12.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1bb80a3bb0 0x7f1bb80a6070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd4104530 0x7f1bd41a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd4103b70 0x7f1bd4199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 --2- 172.21.15.94:0/837880147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd4102980 0x7f1bd4198bd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.735+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 >> 172.21.15.94:0/837880147 conn(0x7f1bd40fdf10 msgr2=0x7f1bd4106f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:12.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.736+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 shutdown_connections 2024-04-26T20:09:12.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.736+0000 7f1bdb200640 1 -- 172.21.15.94:0/837880147 wait complete. 2024-04-26T20:09:12.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:09:12.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.963+0000 7fa31b200640 1 -- 172.21.15.94:0/4019819273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c104a70 msgr2=0x7fa31c104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.963+0000 7fa31b200640 1 --2- 172.21.15.94:0/4019819273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c104a70 0x7fa31c104e90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009900 tx=0x7fa30c02a410 comp rx=0 tx=0).stop 2024-04-26T20:09:12.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.963+0000 7fa31b200640 1 -- 172.21.15.94:0/4019819273 shutdown_connections 2024-04-26T20:09:12.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.963+0000 7fa31b200640 1 --2- 172.21.15.94:0/4019819273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa31c106620 0x7fa31c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.963+0000 7fa31b200640 1 --2- 172.21.15.94:0/4019819273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c105c60 0x7fa31c1060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.963+0000 7fa31b200640 1 --2- 172.21.15.94:0/4019819273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c104a70 0x7fa31c104e90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.963+0000 7fa31b200640 1 -- 172.21.15.94:0/4019819273 >> 172.21.15.94:0/4019819273 conn(0x7fa31c0fffe0 msgr2=0x7fa31c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:12.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.963+0000 7fa31b200640 1 -- 172.21.15.94:0/4019819273 shutdown_connections 2024-04-26T20:09:12.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.963+0000 7fa31b200640 1 -- 172.21.15.94:0/4019819273 wait complete. 2024-04-26T20:09:12.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.964+0000 7fa31b200640 1 Processor -- start 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.964+0000 7fa31b200640 1 -- start start 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa31c104a70 0x7fa31c19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c105c60 0x7fa31c19b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31b200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c106620 0x7fa31c1a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31b200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c1105f0 con 0x7fa31c106620 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c110470 con 0x7fa31c105c60 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c110770 con 0x7fa31c104a70 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c106620 0x7fa31c1a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c106620 0x7fa31c1a2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34544/0 (socket says 172.21.15.94:34544) 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31a800640 1 -- 172.21.15.94:0/778451496 learned_addr learned my addr 172.21.15.94:0/778451496 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa319e00640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa31c104a70 0x7fa31c19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa313e00640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c105c60 0x7fa31c19b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31a800640 1 -- 172.21.15.94:0/778451496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa31c104a70 msgr2=0x7fa31c19ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31a800640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa31c104a70 0x7fa31c19ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31a800640 1 -- 172.21.15.94:0/778451496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c105c60 msgr2=0x7fa31c19b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:12.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31a800640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c105c60 0x7fa31c19b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:12.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.965+0000 7fa31a800640 1 -- 172.21.15.94:0/778451496 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31c1a28d0 con 0x7fa31c106620 2024-04-26T20:09:12.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.966+0000 7fa31a800640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c106620 0x7fa31c1a2230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa31400eb10 tx=0x7fa31400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:12.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.966+0000 7fa312a00640 1 -- 172.21.15.94:0/778451496 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314004b20 con 0x7fa31c106620 2024-04-26T20:09:12.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.966+0000 7fa312a00640 1 -- 172.21.15.94:0/778451496 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa314004cc0 con 0x7fa31c106620 2024-04-26T20:09:12.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.966+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31c1a2bc0 con 0x7fa31c106620 2024-04-26T20:09:12.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.966+0000 7fa312a00640 1 -- 172.21.15.94:0/778451496 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314010670 con 0x7fa31c106620 2024-04-26T20:09:12.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.966+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa31c1a30e0 con 0x7fa31c106620 2024-04-26T20:09:12.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.967+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa31c10be10 con 0x7fa31c106620 2024-04-26T20:09:12.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.969+0000 7fa312a00640 1 -- 172.21.15.94:0/778451496 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa314005020 con 0x7fa31c106620 2024-04-26T20:09:12.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.970+0000 7fa312a00640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa3000a3bb0 0x7fa3000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:12.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.971+0000 7fa312a00640 1 -- 172.21.15.94:0/778451496 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa3140d0ce0 con 0x7fa31c106620 2024-04-26T20:09:12.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.971+0000 7fa319e00640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa3000a3bb0 0x7fa3000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:12.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.971+0000 7fa319e00640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa3000a3bb0 0x7fa3000a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa30c005ab0 tx=0x7fa30c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:12.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:12.975+0000 7fa312a00640 1 -- 172.21.15.94:0/778451496 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa31409d300 con 0x7fa31c106620 2024-04-26T20:09:13.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.178+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa31c063530 con 0x7fa31c106620 2024-04-26T20:09:13.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.179+0000 7fa312a00640 1 -- 172.21.15.94:0/778451496 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fa31409cc00 con 0x7fa31c106620 2024-04-26T20:09:13.180 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:09:13.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa3000a3bb0 msgr2=0x7fa3000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa3000a3bb0 0x7fa3000a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa30c005ab0 tx=0x7fa30c005a20 comp rx=0 tx=0).stop 2024-04-26T20:09:13.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c106620 msgr2=0x7fa31c1a2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c106620 0x7fa31c1a2230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa31400eb10 tx=0x7fa31400c2d0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 shutdown_connections 2024-04-26T20:09:13.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa3000a3bb0 0x7fa3000a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa31c106620 0x7fa31c1a2230 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c105c60 0x7fa31c19b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 --2- 172.21.15.94:0/778451496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa31c104a70 0x7fa31c19ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 >> 172.21.15.94:0/778451496 conn(0x7fa31c0fffe0 msgr2=0x7fa31c108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:13.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 shutdown_connections 2024-04-26T20:09:13.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.182+0000 7fa31b200640 1 -- 172.21.15.94:0/778451496 wait complete. 2024-04-26T20:09:13.203 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:09:13.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.411+0000 7f1667c00640 1 -- 172.21.15.94:0/840332594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1660106740 msgr2=0x7f166010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.411+0000 7f1667c00640 1 --2- 172.21.15.94:0/840332594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1660106740 0x7f166010cfd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1658009be0 tx=0x7f165802a6d0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.411+0000 7f1667c00640 1 -- 172.21.15.94:0/840332594 shutdown_connections 2024-04-26T20:09:13.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.411+0000 7f1667c00640 1 --2- 172.21.15.94:0/840332594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1660106740 0x7f166010cfd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.411+0000 7f1667c00640 1 --2- 172.21.15.94:0/840332594 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1660105d80 0x7f1660106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.411+0000 7f1667c00640 1 --2- 172.21.15.94:0/840332594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1660104b90 0x7f1660104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.411+0000 7f1667c00640 1 -- 172.21.15.94:0/840332594 >> 172.21.15.94:0/840332594 conn(0x7f1660100120 msgr2=0x7f1660102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:13.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.411+0000 7f1667c00640 1 -- 172.21.15.94:0/840332594 shutdown_connections 2024-04-26T20:09:13.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.411+0000 7f1667c00640 1 -- 172.21.15.94:0/840332594 wait complete. 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.412+0000 7f1667c00640 1 Processor -- start 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.412+0000 7f1667c00640 1 -- start start 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f1667c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1660104b90 0x7f166019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f1667c00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1660105d80 0x7f166019b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f1667c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1660106740 0x7f16601a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f1667c00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f16601106a0 con 0x7f1660105d80 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f1667c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1660110520 con 0x7f1660106740 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f1667c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1660110820 con 0x7f1660104b90 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f165fe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1660105d80 0x7f166019b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f1665c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1660106740 0x7f16601a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f165fe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1660105d80 0x7f166019b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34566/0 (socket says 172.21.15.94:34566) 2024-04-26T20:09:13.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f165fe00640 1 -- 172.21.15.94:0/3481658223 learned_addr learned my addr 172.21.15.94:0/3481658223 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.413+0000 7f1665200640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1660104b90 0x7f166019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f1665c00640 1 -- 172.21.15.94:0/3481658223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1660104b90 msgr2=0x7f166019ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f1665c00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1660104b90 0x7f166019ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f1665c00640 1 -- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1660105d80 msgr2=0x7f166019b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f1665c00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1660105d80 0x7f166019b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f1665c00640 1 -- 172.21.15.94:0/3481658223 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16601a2920 con 0x7f1660106740 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f165fe00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1660105d80 0x7f166019b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f1665c00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1660106740 0x7f16601a2280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1658034040 tx=0x7f1658030e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f165d600640 1 -- 172.21.15.94:0/3481658223 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658041020 con 0x7f1660106740 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f165d600640 1 -- 172.21.15.94:0/3481658223 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16580318f0 con 0x7f1660106740 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16601a2bb0 con 0x7f1660106740 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f165d600640 1 -- 172.21.15.94:0/3481658223 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658036040 con 0x7f1660106740 2024-04-26T20:09:13.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.414+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16601a3070 con 0x7f1660106740 2024-04-26T20:09:13.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.415+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166010beb0 con 0x7f1660106740 2024-04-26T20:09:13.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.418+0000 7f165d600640 1 -- 172.21.15.94:0/3481658223 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1658044020 con 0x7f1660106740 2024-04-26T20:09:13.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.419+0000 7f165d600640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16480a3bb0 0x7f16480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:13.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.419+0000 7f165d600640 1 -- 172.21.15.94:0/3481658223 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f16580f1950 con 0x7f1660106740 2024-04-26T20:09:13.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.419+0000 7f1665200640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16480a3bb0 0x7f16480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:13.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.420+0000 7f1665200640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16480a3bb0 0x7f16480a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f166019c1e0 tx=0x7f165400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:13.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.421+0000 7f165d600640 1 -- 172.21.15.94:0/3481658223 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f165804b020 con 0x7f1660106740 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T20:09:13.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:09:13.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:09:13.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:13.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/837880147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:09:13.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:13 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/778451496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:09:13.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2024-04-26T20:09:13.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-26T20:09:13.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/837880147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:09:13.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:13 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/778451496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:09:13.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.622+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f166002dd00 con 0x7f1660106740 2024-04-26T20:09:13.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.623+0000 7f165d600640 1 -- 172.21.15.94:0/3481658223 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f16580bdf70 con 0x7f1660106740 2024-04-26T20:09:13.624 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.625+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16480a3bb0 msgr2=0x7f16480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.625+0000 7f1667c00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16480a3bb0 0x7f16480a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f166019c1e0 tx=0x7f165400b390 comp rx=0 tx=0).stop 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.625+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1660106740 msgr2=0x7f16601a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.625+0000 7f1667c00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1660106740 0x7f16601a2280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1658034040 tx=0x7f1658030e00 comp rx=0 tx=0).stop 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.626+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 shutdown_connections 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.626+0000 7f1667c00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f16480a3bb0 0x7f16480a6070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.626+0000 7f1667c00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1660106740 0x7f16601a2280 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.626+0000 7f1667c00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1660105d80 0x7f166019b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.626+0000 7f1667c00640 1 --2- 172.21.15.94:0/3481658223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1660104b90 0x7f166019ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.626+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 >> 172.21.15.94:0/3481658223 conn(0x7f1660100120 msgr2=0x7f1660109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:13.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.626+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 shutdown_connections 2024-04-26T20:09:13.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.626+0000 7f1667c00640 1 -- 172.21.15.94:0/3481658223 wait complete. 2024-04-26T20:09:13.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:09:13.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/837880147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:09:13.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:13 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/778451496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:09:13.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:09:13.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.829+0000 7fe2bae00640 1 -- 172.21.15.94:0/3677382854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4102980 msgr2=0x7fe2b4102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.829+0000 7fe2bae00640 1 --2- 172.21.15.94:0/3677382854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4102980 0x7fe2b4102da0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009900 tx=0x7fe2a402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.829+0000 7fe2bae00640 1 -- 172.21.15.94:0/3677382854 shutdown_connections 2024-04-26T20:09:13.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.829+0000 7fe2bae00640 1 --2- 172.21.15.94:0/3677382854 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2b4104530 0x7fe2b410adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.829+0000 7fe2bae00640 1 --2- 172.21.15.94:0/3677382854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4103b70 0x7fe2b4103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.829+0000 7fe2bae00640 1 --2- 172.21.15.94:0/3677382854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4102980 0x7fe2b4102da0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.829+0000 7fe2bae00640 1 -- 172.21.15.94:0/3677382854 >> 172.21.15.94:0/3677382854 conn(0x7fe2b40fdf10 msgr2=0x7fe2b4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:13.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.829+0000 7fe2bae00640 1 -- 172.21.15.94:0/3677382854 shutdown_connections 2024-04-26T20:09:13.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.830+0000 7fe2bae00640 1 -- 172.21.15.94:0/3677382854 wait complete. 2024-04-26T20:09:13.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.830+0000 7fe2bae00640 1 Processor -- start 2024-04-26T20:09:13.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2bae00640 1 -- start start 2024-04-26T20:09:13.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2bae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4102980 0x7fe2b419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:13.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2bae00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2b4103b70 0x7fe2b419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:13.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2bae00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4104530 0x7fe2b41a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:13.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2bae00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b410e480 con 0x7fe2b4104530 2024-04-26T20:09:13.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2bae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b410e300 con 0x7fe2b4102980 2024-04-26T20:09:13.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2bae00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b410e600 con 0x7fe2b4103b70 2024-04-26T20:09:13.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2b8e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4104530 0x7fe2b41a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:13.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2b3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2b4103b70 0x7fe2b419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2b8e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4104530 0x7fe2b41a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34570/0 (socket says 172.21.15.94:34570) 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2b8e00640 1 -- 172.21.15.94:0/2595748421 learned_addr learned my addr 172.21.15.94:0/2595748421 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.831+0000 7fe2b3e00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4102980 0x7fe2b419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b3400640 1 -- 172.21.15.94:0/2595748421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4102980 msgr2=0x7fe2b419ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b3400640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4102980 0x7fe2b419ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b3400640 1 -- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4104530 msgr2=0x7fe2b41a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b3400640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4104530 0x7fe2b41a23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b3400640 1 -- 172.21.15.94:0/2595748421 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b41a2a40 con 0x7fe2b4103b70 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b8e00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4104530 0x7fe2b41a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b3400640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2b4103b70 0x7fe2b419b360 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000d6d0 tx=0x7fe2a000db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b0c00640 1 -- 172.21.15.94:0/2595748421 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0017070 con 0x7fe2b4103b70 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b41a2d30 con 0x7fe2b4103b70 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b0c00640 1 -- 172.21.15.94:0/2595748421 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2a0009820 con 0x7fe2b4103b70 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2b0c00640 1 -- 172.21.15.94:0/2595748421 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0004c40 con 0x7fe2b4103b70 2024-04-26T20:09:13.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.832+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b41a3220 con 0x7fe2b4103b70 2024-04-26T20:09:13.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.833+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b41a3450 con 0x7fe2b4103b70 2024-04-26T20:09:13.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.836+0000 7fe2b0c00640 1 -- 172.21.15.94:0/2595748421 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe2a0005220 con 0x7fe2b4103b70 2024-04-26T20:09:13.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.837+0000 7fe2b0c00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe2980a3bb0 0x7fe2980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:13.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.837+0000 7fe2b0c00640 1 -- 172.21.15.94:0/2595748421 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fe2a00d0640 con 0x7fe2b4103b70 2024-04-26T20:09:13.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.837+0000 7fe2b3e00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe2980a3bb0 0x7fe2980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:13.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.838+0000 7fe2b3e00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe2980a3bb0 0x7fe2980a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009230 tx=0x7fe2a40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:13.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:13.839+0000 7fe2b0c00640 1 -- 172.21.15.94:0/2595748421 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe2a009c610 con 0x7fe2b4103b70 2024-04-26T20:09:14.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.043+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe2b41a3620 con 0x7fe2b4103b70 2024-04-26T20:09:14.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.044+0000 7fe2b0c00640 1 -- 172.21.15.94:0/2595748421 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fe2a00a0910 con 0x7fe2b4103b70 2024-04-26T20:09:14.045 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:09:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe2980a3bb0 msgr2=0x7fe2980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe2980a3bb0 0x7fe2980a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009230 tx=0x7fe2a40314a0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2b4103b70 msgr2=0x7fe2b419b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2b4103b70 0x7fe2b419b360 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000d6d0 tx=0x7fe2a000db90 comp rx=0 tx=0).stop 2024-04-26T20:09:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 shutdown_connections 2024-04-26T20:09:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe2980a3bb0 0x7fe2980a6070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2b4104530 0x7fe2b41a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2b4103b70 0x7fe2b419b360 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 --2- 172.21.15.94:0/2595748421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2b4102980 0x7fe2b419ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 >> 172.21.15.94:0/2595748421 conn(0x7fe2b40fdf10 msgr2=0x7fe2b4109330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:14.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 shutdown_connections 2024-04-26T20:09:14.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.047+0000 7fe2bae00640 1 -- 172.21.15.94:0/2595748421 wait complete. 2024-04-26T20:09:14.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T20:09:14.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.263+0000 7f9dab200640 1 -- 172.21.15.94:0/762188684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac103960 msgr2=0x7f9dac103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:14.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.263+0000 7f9dab200640 1 --2- 172.21.15.94:0/762188684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac103960 0x7f9dac103de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009900 tx=0x7f9d9402a410 comp rx=0 tx=0).stop 2024-04-26T20:09:14.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.263+0000 7f9dab200640 1 -- 172.21.15.94:0/762188684 shutdown_connections 2024-04-26T20:09:14.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.263+0000 7f9dab200640 1 --2- 172.21.15.94:0/762188684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac104320 0x7f9dac10abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.263+0000 7f9dab200640 1 --2- 172.21.15.94:0/762188684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac103960 0x7f9dac103de0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.263+0000 7f9dab200640 1 --2- 172.21.15.94:0/762188684 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dac102770 0x7f9dac102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.263+0000 7f9dab200640 1 -- 172.21.15.94:0/762188684 >> 172.21.15.94:0/762188684 conn(0x7f9dac0fdce0 msgr2=0x7f9dac100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:14.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.263+0000 7f9dab200640 1 -- 172.21.15.94:0/762188684 shutdown_connections 2024-04-26T20:09:14.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.263+0000 7f9dab200640 1 -- 172.21.15.94:0/762188684 wait complete. 2024-04-26T20:09:14.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.264+0000 7f9dab200640 1 Processor -- start 2024-04-26T20:09:14.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.264+0000 7f9dab200640 1 -- start start 2024-04-26T20:09:14.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.264+0000 7f9dab200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac102770 0x7f9dac19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:14.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.264+0000 7f9dab200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dac103960 0x7f9dac19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:14.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.264+0000 7f9dab200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac104320 0x7f9dac1a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:14.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.264+0000 7f9dab200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac10e2a0 con 0x7f9dac104320 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.264+0000 7f9dab200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac10e120 con 0x7f9dac102770 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.264+0000 7f9dab200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac10e420 con 0x7f9dac103960 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac102770 0x7f9dac19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac102770 0x7f9dac19ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:41778/0 (socket says 172.21.15.94:41778) 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dac103960 0x7f9dac19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da9e00640 1 -- 172.21.15.94:0/1794372537 learned_addr learned my addr 172.21.15.94:0/1794372537 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9daa800640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac104320 0x7f9dac1a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da9e00640 1 -- 172.21.15.94:0/1794372537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dac103960 msgr2=0x7f9dac19b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da9e00640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dac103960 0x7f9dac19b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da9e00640 1 -- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac104320 msgr2=0x7f9dac1a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da9e00640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac104320 0x7f9dac1a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da9e00640 1 -- 172.21.15.94:0/1794372537 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dac1a27a0 con 0x7f9dac102770 2024-04-26T20:09:14.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.265+0000 7f9da9e00640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac102770 0x7f9dac19ab80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00e9c0 tx=0x7f9d9c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:14.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.266+0000 7f9da2a00640 1 -- 172.21.15.94:0/1794372537 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c00cd80 con 0x7f9dac102770 2024-04-26T20:09:14.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.266+0000 7f9da2a00640 1 -- 172.21.15.94:0/1794372537 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d9c0035f0 con 0x7f9dac102770 2024-04-26T20:09:14.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.266+0000 7f9da2a00640 1 -- 172.21.15.94:0/1794372537 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c010690 con 0x7f9dac102770 2024-04-26T20:09:14.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.266+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dac1a2a90 con 0x7f9dac102770 2024-04-26T20:09:14.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.266+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dac1a2fb0 con 0x7f9dac102770 2024-04-26T20:09:14.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.267+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dac1a3240 con 0x7f9dac102770 2024-04-26T20:09:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.273+0000 7f9da2a00640 1 -- 172.21.15.94:0/1794372537 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9d9c004000 con 0x7f9dac102770 2024-04-26T20:09:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.274+0000 7f9da2a00640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9d900a3c30 0x7f9d900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.274+0000 7f9da2a00640 1 -- 172.21.15.94:0/1794372537 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f9d9c0d0df0 con 0x7f9dac102770 2024-04-26T20:09:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.274+0000 7f9da3e00640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9d900a3c30 0x7f9d900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:14.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.275+0000 7f9da3e00640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9d900a3c30 0x7f9d900a60f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d94000c00 tx=0x7f9d940314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:14.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.275+0000 7f9da2a00640 1 -- 172.21.15.94:0/1794372537 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9d9c09d410 con 0x7f9dac102770 2024-04-26T20:09:14.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.473+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9dac1a34b0 con 0x7f9dac102770 2024-04-26T20:09:14.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.474+0000 7f9da2a00640 1 -- 172.21.15.94:0/1794372537 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v94) ==== 81+0+591 (secure 0 0 0) 0x7f9d9c09cd10 con 0x7f9dac102770 2024-04-26T20:09:14.504 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3877259400 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:14.511 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3361956945 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:14.517 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3651438916 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:14.588 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3005317891 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:14.619 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1688404742 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:14.700 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/2041870658 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:14.755 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/442392134 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:14.855 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1853030572 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:14.863 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3005317891 2024-04-27T19:59:38.717857+0000 2024-04-26T20:09:14.874 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3651438916 2024-04-27T20:00:11.413271+0000 2024-04-26T20:09:14.883 INFO:teuthology.orchestra.run.smithi094.stderr:listed 10 entries 2024-04-26T20:09:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9d900a3c30 msgr2=0x7f9d900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:14.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9d900a3c30 0x7f9d900a60f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d94000c00 tx=0x7f9d940314a0 comp rx=0 tx=0).stop 2024-04-26T20:09:14.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac102770 msgr2=0x7f9dac19ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:14.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac102770 0x7f9dac19ab80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00e9c0 tx=0x7f9d9c00ee80 comp rx=0 tx=0).stop 2024-04-26T20:09:15.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 shutdown_connections 2024-04-26T20:09:15.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9d900a3c30 0x7f9d900a60f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:15.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac104320 0x7f9dac1a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:15.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dac103960 0x7f9dac19b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:15.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 --2- 172.21.15.94:0/1794372537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9dac102770 0x7f9dac19ab80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:15.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 >> 172.21.15.94:0/1794372537 conn(0x7f9dac0fdce0 msgr2=0x7f9dac074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:15.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 shutdown_connections 2024-04-26T20:09:15.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:14.477+0000 7f9dab200640 1 -- 172.21.15.94:0/1794372537 wait complete. 2024-04-26T20:09:15.279 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:14 smithi094 ceph-mon[28050]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T20:09:15.327 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:14 smithi094 ceph-mon[28050]: pgmap v355: 129 pgs: 7 peering, 122 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:15.336 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:14 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3481658223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:09:15.345 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:14 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2595748421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:09:15.354 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi094.front.sepia.ceph.com... 2024-04-26T20:09:15.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:09:15.373 DEBUG:teuthology.orchestra.run.smithi094:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:09:15.383 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:14 smithi204 ceph-mon[32618]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T20:09:15.465 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:14 smithi204 ceph-mon[32618]: pgmap v355: 129 pgs: 7 peering, 122 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:15.474 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:14 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3481658223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:09:15.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:14 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2595748421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:09:15.491 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:14 smithi103 ceph-mon[32635]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T20:09:15.558 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:14 smithi103 ceph-mon[32635]: pgmap v355: 129 pgs: 7 peering, 122 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:15.573 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:14 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3481658223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:09:15.582 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:14 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2595748421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:09:15.594 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:15 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1794372537' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:09:15.669 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:15 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1794372537' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:09:15.734 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:15 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1794372537' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:09:15.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:ceph-fuse[68972]: fuse finished with error 0 and tester_r 0 2024-04-26T20:09:15.783 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T20:09:15.814 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi094.front.sepia.ceph.com 2024-04-26T20:09:15.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:09:15.829 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-26T20:09:15.931 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:09:16.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.136+0000 7f1039600640 1 -- 172.21.15.94:0/3209965408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1034102750 msgr2=0x7f1034102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:16.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.136+0000 7f1039600640 1 --2- 172.21.15.94:0/3209965408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1034102750 0x7f1034102b70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1024009900 tx=0x7f102402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:09:16.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.136+0000 7f1039600640 1 -- 172.21.15.94:0/3209965408 shutdown_connections 2024-04-26T20:09:16.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.136+0000 7f1039600640 1 --2- 172.21.15.94:0/3209965408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1034104300 0x7f103410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:16.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.136+0000 7f1039600640 1 --2- 172.21.15.94:0/3209965408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1034103940 0x7f1034103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:16.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.136+0000 7f1039600640 1 --2- 172.21.15.94:0/3209965408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1034102750 0x7f1034102b70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:16.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.136+0000 7f1039600640 1 -- 172.21.15.94:0/3209965408 >> 172.21.15.94:0/3209965408 conn(0x7f10340fdcc0 msgr2=0x7f1034100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:16.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.136+0000 7f1039600640 1 -- 172.21.15.94:0/3209965408 shutdown_connections 2024-04-26T20:09:16.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.136+0000 7f1039600640 1 -- 172.21.15.94:0/3209965408 wait complete. 2024-04-26T20:09:16.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.137+0000 7f1039600640 1 Processor -- start 2024-04-26T20:09:16.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.137+0000 7f1039600640 1 -- start start 2024-04-26T20:09:16.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.137+0000 7f1039600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1034102750 0x7f10341989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:16.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.137+0000 7f1039600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1034103940 0x7f1034198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:16.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.137+0000 7f1039600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1034104300 0x7f103419ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:16.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.137+0000 7f1039600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f103410e2a0 con 0x7f1034103940 2024-04-26T20:09:16.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.137+0000 7f1039600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f103410e120 con 0x7f1034102750 2024-04-26T20:09:16.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.137+0000 7f1039600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f103410e420 con 0x7f1034104300 2024-04-26T20:09:16.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1034103940 0x7f1034198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:16.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1034103940 0x7f1034198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34596/0 (socket says 172.21.15.94:34596) 2024-04-26T20:09:16.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1034102750 0x7f10341989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:16.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033400640 1 -- 172.21.15.94:0/1794309873 learned_addr learned my addr 172.21.15.94:0/1794309873 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:16.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1038c00640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1034104300 0x7f103419ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:16.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033400640 1 -- 172.21.15.94:0/1794309873 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1034104300 msgr2=0x7f103419ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033400640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1034104300 0x7f103419ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:16.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033400640 1 -- 172.21.15.94:0/1794309873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1034102750 msgr2=0x7f10341989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:16.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033400640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1034102750 0x7f10341989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:16.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033400640 1 -- 172.21.15.94:0/1794309873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10341a05c0 con 0x7f1034103940 2024-04-26T20:09:16.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1033400640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1034103940 0x7f1034198ee0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f102000ea60 tx=0x7f102000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:17.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.138+0000 7f1030c00640 1 -- 172.21.15.94:0/1794309873 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102000cde0 con 0x7f1034103940 2024-04-26T20:09:17.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.139+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10341a08b0 con 0x7f1034103940 2024-04-26T20:09:17.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.139+0000 7f1030c00640 1 -- 172.21.15.94:0/1794309873 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10200035f0 con 0x7f1034103940 2024-04-26T20:09:17.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.139+0000 7f1030c00640 1 -- 172.21.15.94:0/1794309873 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1020005220 con 0x7f1034103940 2024-04-26T20:09:17.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.139+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10341a0e50 con 0x7f1034103940 2024-04-26T20:09:17.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.140+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1034109ae0 con 0x7f1034103940 2024-04-26T20:09:17.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.142+0000 7f1030c00640 1 -- 172.21.15.94:0/1794309873 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f10200037a0 con 0x7f1034103940 2024-04-26T20:09:17.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.144+0000 7f1030c00640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f10180a3bb0 0x7f10180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:17.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.144+0000 7f1030c00640 1 -- 172.21.15.94:0/1794309873 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f10200d09d0 con 0x7f1034103940 2024-04-26T20:09:17.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.144+0000 7f1033e00640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f10180a3bb0 0x7f10180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:17.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.145+0000 7f1033e00640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f10180a3bb0 0x7f10180a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1024005ab0 tx=0x7f1024005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:17.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.148+0000 7f1030c00640 1 -- 172.21.15.94:0/1794309873 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f102009cff0 con 0x7f1034103940 2024-04-26T20:09:17.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.351+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1034063530 con 0x7f1034103940 2024-04-26T20:09:17.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.354+0000 7f1030c00640 1 -- 172.21.15.94:0/1794309873 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24635 (secure 0 0 0) 0x7f102009c8f0 con 0x7f1034103940 2024-04-26T20:09:17.262 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:16 smithi204 ceph-mon[32618]: pgmap v356: 129 pgs: 7 peering, 122 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:17.314 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:16 smithi103 ceph-mon[32635]: pgmap v356: 129 pgs: 7 peering, 122 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:17.365 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:17.404 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:09:13.256835+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:09:17.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f10180a3bb0 msgr2=0x7f10180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:17.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f10180a3bb0 0x7f10180a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1024005ab0 tx=0x7f1024005a20 comp rx=0 tx=0).stop 2024-04-26T20:09:17.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1034103940 msgr2=0x7f1034198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:17.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1034103940 0x7f1034198ee0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f102000ea60 tx=0x7f102000ef20 comp rx=0 tx=0).stop 2024-04-26T20:09:17.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 shutdown_connections 2024-04-26T20:09:17.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f10180a3bb0 0x7f10180a6070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:17.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1034104300 0x7f103419ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:17.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1034103940 0x7f1034198ee0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 --2- 172.21.15.94:0/1794309873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1034102750 0x7f10341989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:17.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 >> 172.21.15.94:0/1794309873 conn(0x7f10340fdcc0 msgr2=0x7f1034106cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:17.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 shutdown_connections 2024-04-26T20:09:17.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:16.358+0000 7f1039600640 1 -- 172.21.15.94:0/1794309873 wait complete. 2024-04-26T20:09:17.654 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:09:17.661 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:16 smithi094 ceph-mon[28050]: pgmap v356: 129 pgs: 7 peering, 122 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:17.707 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:17 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1794309873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:17.769 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:17 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1794309873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:17.807 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:17 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1794309873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:17.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.913+0000 7f2a72600640 1 -- 172.21.15.94:0/3618900646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a6c108d90 msgr2=0x7f2a6c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:17.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.913+0000 7f2a72600640 1 --2- 172.21.15.94:0/3618900646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a6c108d90 0x7f2a6c1091b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009900 tx=0x7f2a5c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:09:17.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.914+0000 7f2a72600640 1 -- 172.21.15.94:0/3618900646 shutdown_connections 2024-04-26T20:09:18.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.914+0000 7f2a72600640 1 --2- 172.21.15.94:0/3618900646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a6c10a940 0x7f2a6c1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:18.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.914+0000 7f2a72600640 1 --2- 172.21.15.94:0/3618900646 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a6c109f80 0x7f2a6c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:18.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.914+0000 7f2a72600640 1 --2- 172.21.15.94:0/3618900646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a6c108d90 0x7f2a6c1091b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.914+0000 7f2a72600640 1 -- 172.21.15.94:0/3618900646 >> 172.21.15.94:0/3618900646 conn(0x7f2a6c077d30 msgr2=0x7f2a6c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:18.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.914+0000 7f2a72600640 1 -- 172.21.15.94:0/3618900646 shutdown_connections 2024-04-26T20:09:18.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.914+0000 7f2a72600640 1 -- 172.21.15.94:0/3618900646 wait complete. 2024-04-26T20:09:18.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.915+0000 7f2a72600640 1 Processor -- start 2024-04-26T20:09:18.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.915+0000 7f2a72600640 1 -- start start 2024-04-26T20:09:18.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.915+0000 7f2a72600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a6c108d90 0x7f2a6c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:18.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.915+0000 7f2a72600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a6c109f80 0x7f2a6c19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.915+0000 7f2a72600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a6c10a940 0x7f2a6c1a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:18.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.915+0000 7f2a72600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c114890 con 0x7f2a6c109f80 2024-04-26T20:09:18.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.915+0000 7f2a72600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c114710 con 0x7f2a6c108d90 2024-04-26T20:09:18.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.915+0000 7f2a72600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c114a10 con 0x7f2a6c10a940 2024-04-26T20:09:18.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a6c10a940 0x7f2a6c1a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:18.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a6c108d90 0x7f2a6c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:18.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a6c10a940 0x7f2a6c1a6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:48342/0 (socket says 172.21.15.94:48342) 2024-04-26T20:09:18.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71c00640 1 -- 172.21.15.94:0/2269011178 learned_addr learned my addr 172.21.15.94:0/2269011178 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:18.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a63e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a6c109f80 0x7f2a6c19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:18.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71c00640 1 -- 172.21.15.94:0/2269011178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a6c108d90 msgr2=0x7f2a6c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:18.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71c00640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a6c108d90 0x7f2a6c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:18.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71c00640 1 -- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a6c109f80 msgr2=0x7f2a6c19f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:18.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71c00640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a6c109f80 0x7f2a6c19f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:18.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71c00640 1 -- 172.21.15.94:0/2269011178 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c1a6bc0 con 0x7f2a6c10a940 2024-04-26T20:09:18.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a63e00640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a6c109f80 0x7f2a6c19f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:09:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.916+0000 7f2a71c00640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a6c10a940 0x7f2a6c1a6550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400eb10 tx=0x7f2a6400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:18.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.917+0000 7f2a6a000640 1 -- 172.21.15.94:0/2269011178 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64004b20 con 0x7f2a6c10a940 2024-04-26T20:09:18.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.917+0000 7f2a6a000640 1 -- 172.21.15.94:0/2269011178 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a64004cc0 con 0x7f2a6c10a940 2024-04-26T20:09:18.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.917+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6c1a6eb0 con 0x7f2a6c10a940 2024-04-26T20:09:18.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.917+0000 7f2a6a000640 1 -- 172.21.15.94:0/2269011178 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64010670 con 0x7f2a6c10a940 2024-04-26T20:09:18.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.917+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a6c1a7370 con 0x7f2a6c10a940 2024-04-26T20:09:18.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.918+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a6c110180 con 0x7f2a6c10a940 2024-04-26T20:09:18.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.921+0000 7f2a6a000640 1 -- 172.21.15.94:0/2269011178 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2a64005020 con 0x7f2a6c10a940 2024-04-26T20:09:19.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.922+0000 7f2a6a000640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2a500a3c30 0x7f2a500a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:19.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.922+0000 7f2a6a000640 1 -- 172.21.15.94:0/2269011178 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f2a640d0c50 con 0x7f2a6c10a940 2024-04-26T20:09:19.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.923+0000 7f2a71200640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2a500a3c30 0x7f2a500a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:19.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.923+0000 7f2a71200640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2a500a3c30 0x7f2a500a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009230 tx=0x7f2a5c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:19.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:17.924+0000 7f2a6a000640 1 -- 172.21.15.94:0/2269011178 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2a6409d270 con 0x7f2a6c10a940 2024-04-26T20:09:19.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.127+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a6c02dca0 con 0x7f2a6c10a940 2024-04-26T20:09:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:19.147 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:18 smithi103 ceph-mon[32635]: pgmap v357: 129 pgs: 6 peering, 123 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:19.185 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:18 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2269011178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:19.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:18 smithi204 ceph-mon[32618]: pgmap v357: 129 pgs: 6 peering, 123 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:19.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:18 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2269011178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:19.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.130+0000 7f2a6a000640 1 -- 172.21.15.94:0/2269011178 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24635 (secure 0 0 0) 0x7f2a6409cb70 con 0x7f2a6c10a940 2024-04-26T20:09:19.243 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:09:13.256835+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:09:19.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2a500a3c30 msgr2=0x7f2a500a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:19.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2a500a3c30 0x7f2a500a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009230 tx=0x7f2a5c0314a0 comp rx=0 tx=0).stop 2024-04-26T20:09:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a6c10a940 msgr2=0x7f2a6c1a6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:19.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a6c10a940 0x7f2a6c1a6550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400eb10 tx=0x7f2a6400c2d0 comp rx=0 tx=0).stop 2024-04-26T20:09:19.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 shutdown_connections 2024-04-26T20:09:19.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2a500a3c30 0x7f2a500a60f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:19.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a6c10a940 0x7f2a6c1a6550 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:19.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a6c109f80 0x7f2a6c19f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:19.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 --2- 172.21.15.94:0/2269011178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a6c108d90 0x7f2a6c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 >> 172.21.15.94:0/2269011178 conn(0x7f2a6c077d30 msgr2=0x7f2a6c10d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:19.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 shutdown_connections 2024-04-26T20:09:19.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:18.132+0000 7f2a72600640 1 -- 172.21.15.94:0/2269011178 wait complete. 2024-04-26T20:09:19.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T20:09:19.743 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:18 smithi094 ceph-mon[28050]: pgmap v357: 129 pgs: 6 peering, 123 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:19.754 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:18 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2269011178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:19.761 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:19 smithi094 ceph-mon[28050]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-26T20:09:19.815 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:19 smithi204 ceph-mon[32618]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-26T20:09:19.895 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:19 smithi103 ceph-mon[32635]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-26T20:09:20.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.031+0000 7f2858c00640 1 -- 172.21.15.94:0/3284888597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f285410ee90 msgr2=0x7f2854111280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:20.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.031+0000 7f2858c00640 1 --2- 172.21.15.94:0/3284888597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f285410ee90 0x7f2854111280 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f284c009cd0 tx=0x7f284c02a8f0 comp rx=0 tx=0).stop 2024-04-26T20:09:20.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.031+0000 7f2858c00640 1 -- 172.21.15.94:0/3284888597 shutdown_connections 2024-04-26T20:09:20.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.031+0000 7f2858c00640 1 --2- 172.21.15.94:0/3284888597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f285410ee90 0x7f2854111280 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:20.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.031+0000 7f2858c00640 1 --2- 172.21.15.94:0/3284888597 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2854075390 0x7f2854075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:20.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.031+0000 7f2858c00640 1 --2- 172.21.15.94:0/3284888597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2854076970 0x7f2854074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:20.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.031+0000 7f2858c00640 1 -- 172.21.15.94:0/3284888597 >> 172.21.15.94:0/3284888597 conn(0x7f28540fde30 msgr2=0x7f2854100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:20.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.031+0000 7f2858c00640 1 -- 172.21.15.94:0/3284888597 shutdown_connections 2024-04-26T20:09:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.031+0000 7f2858c00640 1 -- 172.21.15.94:0/3284888597 wait complete. 2024-04-26T20:09:20.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.032+0000 7f2858c00640 1 Processor -- start 2024-04-26T20:09:20.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.032+0000 7f2858c00640 1 -- start start 2024-04-26T20:09:20.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.032+0000 7f2858c00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2854075390 0x7f285419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:20.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.032+0000 7f2858c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2854076970 0x7f285419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:20.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.032+0000 7f2858c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285410ee90 0x7f28541a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:20.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.032+0000 7f2858c00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2854114970 con 0x7f2854075390 2024-04-26T20:09:20.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.032+0000 7f2858c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28541147f0 con 0x7f285410ee90 2024-04-26T20:09:20.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.032+0000 7f2858c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2854114af0 con 0x7f2854076970 2024-04-26T20:09:20.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2854075390 0x7f285419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:20.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285410ee90 0x7f28541a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:20.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2854075390 0x7f285419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34630/0 (socket says 172.21.15.94:34630) 2024-04-26T20:09:20.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853400640 1 -- 172.21.15.94:0/3101309323 learned_addr learned my addr 172.21.15.94:0/3101309323 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:20.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2852a00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2854076970 0x7f285419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:20.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853400640 1 -- 172.21.15.94:0/3101309323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2854076970 msgr2=0x7f285419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:20.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853400640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2854076970 0x7f285419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:20.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853400640 1 -- 172.21.15.94:0/3101309323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285410ee90 msgr2=0x7f28541a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853400640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285410ee90 0x7f28541a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:20.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853400640 1 -- 172.21.15.94:0/3101309323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28541a6c90 con 0x7f2854075390 2024-04-26T20:09:20.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f2853400640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2854075390 0x7f285419f070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f284400ea60 tx=0x7f284400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:20.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.033+0000 7f284be00640 1 -- 172.21.15.94:0/3101309323 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284400cde0 con 0x7f2854075390 2024-04-26T20:09:20.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.034+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28541a6f80 con 0x7f2854075390 2024-04-26T20:09:20.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.034+0000 7f284be00640 1 -- 172.21.15.94:0/3101309323 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28440035f0 con 0x7f2854075390 2024-04-26T20:09:20.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.034+0000 7f284be00640 1 -- 172.21.15.94:0/3101309323 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844010690 con 0x7f2854075390 2024-04-26T20:09:20.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.034+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28541a73e0 con 0x7f2854075390 2024-04-26T20:09:20.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.035+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2854069670 con 0x7f2854075390 2024-04-26T20:09:20.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.037+0000 7f284be00640 1 -- 172.21.15.94:0/3101309323 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f28440037a0 con 0x7f2854075390 2024-04-26T20:09:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.038+0000 7f284be00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f28340a3bb0 0x7f28340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:20.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.038+0000 7f284be00640 1 -- 172.21.15.94:0/3101309323 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f28440d0a50 con 0x7f2854075390 2024-04-26T20:09:20.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.039+0000 7f2852a00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f28340a3bb0 0x7f28340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:21.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.039+0000 7f2852a00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f28340a3bb0 0x7f28340a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f28541a0580 tx=0x7f283c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:21.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.043+0000 7f284be00640 1 -- 172.21.15.94:0/3101309323 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f284409d070 con 0x7f2854075390 2024-04-26T20:09:21.031 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:20 smithi103 ceph-mon[32635]: pgmap v358: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:21.121 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:20 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3101309323' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T20:09:21.155 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:20 smithi204 ceph-mon[32618]: pgmap v358: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:21.165 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:20 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3101309323' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T20:09:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.289+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2854105060 con 0x7f2854075390 2024-04-26T20:09:21.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.304+0000 7f284be00640 1 -- 172.21.15.94:0/3101309323 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f284409c970 con 0x7f2854075390 2024-04-26T20:09:21.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f28340a3bb0 msgr2=0x7f28340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:21.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f28340a3bb0 0x7f28340a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f28541a0580 tx=0x7f283c005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:21.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2854075390 msgr2=0x7f285419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:21.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2854075390 0x7f285419f070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f284400ea60 tx=0x7f284400ef20 comp rx=0 tx=0).stop 2024-04-26T20:09:21.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 shutdown_connections 2024-04-26T20:09:21.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f28340a3bb0 0x7f28340a6070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:21.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f285410ee90 0x7f28541a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:21.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2854076970 0x7f285419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:21.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 --2- 172.21.15.94:0/3101309323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2854075390 0x7f285419f070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 >> 172.21.15.94:0/3101309323 conn(0x7f28540fde30 msgr2=0x7f2854102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:21.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 shutdown_connections 2024-04-26T20:09:21.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:20.308+0000 7f2858c00640 1 -- 172.21.15.94:0/3101309323 wait complete. 2024-04-26T20:09:21.683 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-26T20:09:21.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T20:09:21.700 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:20 smithi094 ceph-mon[28050]: pgmap v358: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:09:21.708 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:20 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3101309323' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T20:09:21.769 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:21 smithi204 ceph-mon[32618]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-26T20:09:21.782 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:21 smithi204 ceph-mon[32618]: Cluster is now healthy 2024-04-26T20:09:21.791 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:21 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3101309323' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T20:09:21.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:21 smithi204 ceph-mon[32618]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:09:21.845 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:21 smithi094 ceph-mon[28050]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-26T20:09:21.904 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:21 smithi094 ceph-mon[28050]: Cluster is now healthy 2024-04-26T20:09:21.947 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:21 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3101309323' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T20:09:21.960 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:21 smithi094 ceph-mon[28050]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:09:22.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:21 smithi103 ceph-mon[32635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-26T20:09:22.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:21 smithi103 ceph-mon[32635]: Cluster is now healthy 2024-04-26T20:09:22.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:21 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3101309323' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T20:09:22.042 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:21 smithi103 ceph-mon[32635]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:09:22.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.983+0000 7fe753200640 1 -- 172.21.15.94:0/1589515534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe75410a980 msgr2=0x7fe7541112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:22.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.983+0000 7fe753200640 1 --2- 172.21.15.94:0/1589515534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe75410a980 0x7fe7541112a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009ca0 tx=0x7fe74c02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:09:22.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.984+0000 7fe753200640 1 -- 172.21.15.94:0/1589515534 shutdown_connections 2024-04-26T20:09:22.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.984+0000 7fe753200640 1 --2- 172.21.15.94:0/1589515534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe75410a980 0x7fe7541112a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.984+0000 7fe753200640 1 --2- 172.21.15.94:0/1589515534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe754109fc0 0x7fe75410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:22.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.984+0000 7fe753200640 1 --2- 172.21.15.94:0/1589515534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe754108e70 0x7fe754109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:22.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.984+0000 7fe753200640 1 -- 172.21.15.94:0/1589515534 >> 172.21.15.94:0/1589515534 conn(0x7fe754077e00 msgr2=0x7fe75407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:22.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.984+0000 7fe753200640 1 -- 172.21.15.94:0/1589515534 shutdown_connections 2024-04-26T20:09:22.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.984+0000 7fe753200640 1 -- 172.21.15.94:0/1589515534 wait complete. 2024-04-26T20:09:22.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.985+0000 7fe753200640 1 Processor -- start 2024-04-26T20:09:22.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.985+0000 7fe753200640 1 -- start start 2024-04-26T20:09:22.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.985+0000 7fe753200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe754108e70 0x7fe75419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.985+0000 7fe753200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe754109fc0 0x7fe75419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:22.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.985+0000 7fe753200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe75410a980 0x7fe7541a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.985+0000 7fe753200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe7541149f0 con 0x7fe754108e70 2024-04-26T20:09:22.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.985+0000 7fe753200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe754114870 con 0x7fe754109fc0 2024-04-26T20:09:22.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.985+0000 7fe753200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe754114b70 con 0x7fe75410a980 2024-04-26T20:09:22.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.985+0000 7fe751e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe754108e70 0x7fe75419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:22.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe751e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe754108e70 0x7fe75419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33730/0 (socket says 172.21.15.94:33730) 2024-04-26T20:09:22.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe751e00640 1 -- 172.21.15.94:0/1912166154 learned_addr learned my addr 172.21.15.94:0/1912166154 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:22.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe749400640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe754109fc0 0x7fe75419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:22.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe752800640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe75410a980 0x7fe7541a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:22.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe751e00640 1 -- 172.21.15.94:0/1912166154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe75410a980 msgr2=0x7fe7541a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe751e00640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe75410a980 0x7fe7541a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:22.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe751e00640 1 -- 172.21.15.94:0/1912166154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe754109fc0 msgr2=0x7fe75419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:22.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe751e00640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe754109fc0 0x7fe75419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:22.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe751e00640 1 -- 172.21.15.94:0/1912166154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7541a6bd0 con 0x7fe754108e70 2024-04-26T20:09:22.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.986+0000 7fe751e00640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe754108e70 0x7fe75419efb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe74400e9f0 tx=0x7fe74400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:22.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.987+0000 7fe74b400640 1 -- 172.21.15.94:0/1912166154 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74400cd80 con 0x7fe754108e70 2024-04-26T20:09:22.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.987+0000 7fe74b400640 1 -- 172.21.15.94:0/1912166154 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7440108e0 con 0x7fe754108e70 2024-04-26T20:09:22.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.987+0000 7fe74b400640 1 -- 172.21.15.94:0/1912166154 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe744018980 con 0x7fe754108e70 2024-04-26T20:09:22.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.987+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7541a6ec0 con 0x7fe754108e70 2024-04-26T20:09:22.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.987+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7541a7320 con 0x7fe754108e70 2024-04-26T20:09:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.988+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe754069af0 con 0x7fe754108e70 2024-04-26T20:09:22.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.990+0000 7fe74b400640 1 -- 172.21.15.94:0/1912166154 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe744010430 con 0x7fe754108e70 2024-04-26T20:09:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.991+0000 7fe74b400640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7380a3bb0 0x7fe7380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:23.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.992+0000 7fe74b400640 1 -- 172.21.15.94:0/1912166154 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fe7440d0a90 con 0x7fe754108e70 2024-04-26T20:09:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.992+0000 7fe749400640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7380a3bb0 0x7fe7380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.993+0000 7fe749400640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7380a3bb0 0x7fe7380a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe7541a04c0 tx=0x7fe73c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:23.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:21.996+0000 7fe74b400640 1 -- 172.21.15.94:0/1912166154 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe754069af0 con 0x7fe754108e70 2024-04-26T20:09:23.206 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:22 smithi103 ceph-mon[32635]: pgmap v359: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 219 B/s wr, 0 op/s 2024-04-26T20:09:23.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:22 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1912166154' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T20:09:23.237 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:22 smithi204 ceph-mon[32618]: pgmap v359: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 219 B/s wr, 0 op/s 2024-04-26T20:09:23.276 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:22 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1912166154' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T20:09:23.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.240+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fe7540635d0 con 0x7fe754108e70 2024-04-26T20:09:23.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.317+0000 7fe74b400640 1 -- 172.21.15.94:0/1912166154 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fe74409ca60 con 0x7fe754108e70 2024-04-26T20:09:23.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7380a3bb0 msgr2=0x7fe7380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:23.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7380a3bb0 0x7fe7380a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe7541a04c0 tx=0x7fe73c005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:23.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe754108e70 msgr2=0x7fe75419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:23.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe754108e70 0x7fe75419efb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe74400e9f0 tx=0x7fe74400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:09:23.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 shutdown_connections 2024-04-26T20:09:23.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7380a3bb0 0x7fe7380a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:23.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe75410a980 0x7fe7541a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:23.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe754109fc0 0x7fe75419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:23.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 --2- 172.21.15.94:0/1912166154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe754108e70 0x7fe75419efb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:23.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.321+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 >> 172.21.15.94:0/1912166154 conn(0x7fe754077e00 msgr2=0x7fe75410d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:23.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.322+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 shutdown_connections 2024-04-26T20:09:23.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:22.322+0000 7fe753200640 1 -- 172.21.15.94:0/1912166154 wait complete. 2024-04-26T20:09:23.760 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-26T20:09:23.770 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:22 smithi094 ceph-mon[28050]: pgmap v359: 129 pgs: 129 active+clean; 581 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 219 B/s wr, 0 op/s 2024-04-26T20:09:23.807 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:22 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1912166154' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T20:09:23.858 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:23 smithi094 ceph-mon[28050]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T20:09:23.896 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1912166154' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T20:09:23.910 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:23 smithi094 ceph-mon[28050]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T20:09:23.925 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:23 smithi094 ceph-mon[28050]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:09:23.939 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:23 smithi094 ceph-mon[28050]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-26T20:09:23.952 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:23 smithi094 ceph-mon[28050]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T20:09:23.966 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:23 smithi103 ceph-mon[32635]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T20:09:23.979 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1912166154' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T20:09:24.047 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:23 smithi103 ceph-mon[32635]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T20:09:24.062 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:23 smithi103 ceph-mon[32635]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:09:24.072 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:23 smithi103 ceph-mon[32635]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-26T20:09:24.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:23 smithi103 ceph-mon[32635]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T20:09:24.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:23 smithi204 ceph-mon[32618]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T20:09:24.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1912166154' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T20:09:24.206 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:23 smithi204 ceph-mon[32618]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T20:09:24.214 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:23 smithi204 ceph-mon[32618]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:09:24.253 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:23 smithi204 ceph-mon[32618]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-26T20:09:24.301 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:23 smithi204 ceph-mon[32618]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T20:09:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.176+0000 7f40b1600640 1 -- 172.21.15.94:0/603421198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 msgr2=0x7f40ac109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:24.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.176+0000 7f40b1600640 1 --2- 172.21.15.94:0/603421198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 0x7f40ac109290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f409c009900 tx=0x7f409c02a410 comp rx=0 tx=0).stop 2024-04-26T20:09:24.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.176+0000 7f40b1600640 1 -- 172.21.15.94:0/603421198 shutdown_connections 2024-04-26T20:09:24.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.176+0000 7f40b1600640 1 --2- 172.21.15.94:0/603421198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40ac10a980 0x7f40ac1112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:24.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.176+0000 7f40b1600640 1 --2- 172.21.15.94:0/603421198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40ac109fc0 0x7f40ac10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:24.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.176+0000 7f40b1600640 1 --2- 172.21.15.94:0/603421198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 0x7f40ac109290 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:24.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.176+0000 7f40b1600640 1 -- 172.21.15.94:0/603421198 >> 172.21.15.94:0/603421198 conn(0x7f40ac077e00 msgr2=0x7f40ac07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.176+0000 7f40b1600640 1 -- 172.21.15.94:0/603421198 shutdown_connections 2024-04-26T20:09:24.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.176+0000 7f40b1600640 1 -- 172.21.15.94:0/603421198 wait complete. 2024-04-26T20:09:24.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.177+0000 7f40b1600640 1 Processor -- start 2024-04-26T20:09:24.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.177+0000 7f40b1600640 1 -- start start 2024-04-26T20:09:24.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.177+0000 7f40b1600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 0x7f40ac19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:24.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.177+0000 7f40b1600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40ac109fc0 0x7f40ac19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:24.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.177+0000 7f40b1600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40ac10a980 0x7f40ac1a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:24.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.177+0000 7f40b1600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac114a10 con 0x7f40ac109fc0 2024-04-26T20:09:24.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.177+0000 7f40b1600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac114890 con 0x7f40ac108e70 2024-04-26T20:09:24.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.177+0000 7f40b1600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac114b90 con 0x7f40ac10a980 2024-04-26T20:09:24.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40abe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 0x7f40ac19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:24.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40abe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 0x7f40ac19f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:50044/0 (socket says 172.21.15.94:50044) 2024-04-26T20:09:24.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40abe00640 1 -- 172.21.15.94:0/382358866 learned_addr learned my addr 172.21.15.94:0/382358866 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:24.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40ab400640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40ac109fc0 0x7f40ac19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:24.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40b0c00640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40ac10a980 0x7f40ac1a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:24.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40abe00640 1 -- 172.21.15.94:0/382358866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40ac10a980 msgr2=0x7f40ac1a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:24.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40abe00640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40ac10a980 0x7f40ac1a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:24.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40abe00640 1 -- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40ac109fc0 msgr2=0x7f40ac19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:24.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40abe00640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40ac109fc0 0x7f40ac19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:24.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40abe00640 1 -- 172.21.15.94:0/382358866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40ac1a6ce0 con 0x7f40ac108e70 2024-04-26T20:09:24.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.178+0000 7f40abe00640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 0x7f40ac19f0c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f409c02aa00 tx=0x7f409c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:25.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.179+0000 7f40a8c00640 1 -- 172.21.15.94:0/382358866 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c03f070 con 0x7f40ac108e70 2024-04-26T20:09:25.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.179+0000 7f40a8c00640 1 -- 172.21.15.94:0/382358866 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f409c031c50 con 0x7f40ac108e70 2024-04-26T20:09:25.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.179+0000 7f40a8c00640 1 -- 172.21.15.94:0/382358866 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c035030 con 0x7f40ac108e70 2024-04-26T20:09:25.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.179+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40ac1a6f70 con 0x7f40ac108e70 2024-04-26T20:09:25.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.179+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40ac1a73d0 con 0x7f40ac108e70 2024-04-26T20:09:25.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.180+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40ac1102a0 con 0x7f40ac108e70 2024-04-26T20:09:25.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.182+0000 7f40a8c00640 1 -- 172.21.15.94:0/382358866 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f409c042070 con 0x7f40ac108e70 2024-04-26T20:09:25.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.183+0000 7f40a8c00640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f40900a3c80 0x7f40900a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:25.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.183+0000 7f40a8c00640 1 -- 172.21.15.94:0/382358866 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f409c0f0f50 con 0x7f40ac108e70 2024-04-26T20:09:25.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.183+0000 7f40ab400640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f40900a3c80 0x7f40900a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.184+0000 7f40ab400640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f40900a3c80 0x7f40900a6140 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f40ac1a05d0 tx=0x7f409800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:25.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.187+0000 7f40a8c00640 1 -- 172.21.15.94:0/382358866 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f409c0bcf80 con 0x7f40ac108e70 2024-04-26T20:09:25.314 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:24 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] up:active 2024-04-26T20:09:25.393 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:24 smithi103 ceph-mon[32635]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T20:09:25.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:24 smithi103 ceph-mon[32635]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T20:09:25.416 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:24 smithi103 ceph-mon[32635]: Cluster is now healthy 2024-04-26T20:09:25.425 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:24 smithi103 ceph-mon[32635]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-26T20:09:25.492 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:24 smithi103 ceph-mon[32635]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-04-26T20:09:25.499 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:24 smithi103 ceph-mon[32635]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T20:09:25.509 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:24 smithi103 ceph-mon[32635]: pgmap v360: 129 pgs: 129 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-26T20:09:25.515 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:24 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] up:active 2024-04-26T20:09:25.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:24 smithi204 ceph-mon[32618]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T20:09:25.626 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:24 smithi204 ceph-mon[32618]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T20:09:25.660 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:24 smithi204 ceph-mon[32618]: Cluster is now healthy 2024-04-26T20:09:25.673 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:24 smithi204 ceph-mon[32618]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-26T20:09:25.683 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:24 smithi204 ceph-mon[32618]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-04-26T20:09:25.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:24 smithi204 ceph-mon[32618]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T20:09:25.710 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:24 smithi204 ceph-mon[32618]: pgmap v360: 129 pgs: 129 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-26T20:09:25.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.434+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f40ac063530 con 0x7f40ac108e70 2024-04-26T20:09:25.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.534+0000 7f40a8c00640 1 -- 172.21.15.94:0/382358866 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f409c0c1280 con 0x7f40ac108e70 2024-04-26T20:09:25.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f40900a3c80 msgr2=0x7f40900a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:25.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f40900a3c80 0x7f40900a6140 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f40ac1a05d0 tx=0x7f409800b390 comp rx=0 tx=0).stop 2024-04-26T20:09:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 msgr2=0x7f40ac19f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:25.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 0x7f40ac19f0c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f409c02aa00 tx=0x7f409c003b90 comp rx=0 tx=0).stop 2024-04-26T20:09:26.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 shutdown_connections 2024-04-26T20:09:26.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f40900a3c80 0x7f40900a6140 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:26.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40ac10a980 0x7f40ac1a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40ac109fc0 0x7f40ac19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 --2- 172.21.15.94:0/382358866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40ac108e70 0x7f40ac19f0c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:26.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 >> 172.21.15.94:0/382358866 conn(0x7f40ac077e00 msgr2=0x7f40ac10d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:26.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 shutdown_connections 2024-04-26T20:09:26.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:24.537+0000 7f40b1600640 1 -- 172.21.15.94:0/382358866 wait complete. 2024-04-26T20:09:26.167 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-26T20:09:26.180 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:24 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] up:active 2024-04-26T20:09:26.194 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:24 smithi094 ceph-mon[28050]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T20:09:26.216 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:24 smithi094 ceph-mon[28050]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T20:09:26.230 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:24 smithi094 ceph-mon[28050]: Cluster is now healthy 2024-04-26T20:09:26.357 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:24 smithi094 ceph-mon[28050]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-26T20:09:26.368 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:24 smithi094 ceph-mon[28050]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-04-26T20:09:26.376 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:24 smithi094 ceph-mon[28050]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T20:09:26.385 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:24 smithi094 ceph-mon[28050]: pgmap v360: 129 pgs: 129 active+clean; 582 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-26T20:09:26.469 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:25 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] up:active 2024-04-26T20:09:26.481 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:25 smithi094 ceph-mon[28050]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-04-26T20:09:26.495 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:25 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/382358866' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:09:26.504 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:25 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:09:26.543 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:25 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T20:09:26.596 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:25 smithi094 ceph-mon[28050]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T20:09:26.634 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:25 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] up:active 2024-04-26T20:09:26.645 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:25 smithi103 ceph-mon[32635]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-04-26T20:09:26.660 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:25 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/382358866' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:09:26.672 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:25 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:09:26.689 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:25 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T20:09:26.696 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:25 smithi103 ceph-mon[32635]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T20:09:26.705 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:26 smithi094 ceph-mon[28050]: pgmap v361: 129 pgs: 129 active+clean; 583 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T20:09:26.788 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:25 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] up:active 2024-04-26T20:09:26.796 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:25 smithi204 ceph-mon[32618]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-04-26T20:09:26.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:25 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/382358866' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:09:26.871 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:25 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:09:26.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:25 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T20:09:26.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:25 smithi204 ceph-mon[32618]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T20:09:26.910 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:26 smithi204 ceph-mon[32618]: pgmap v361: 129 pgs: 129 active+clean; 583 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T20:09:26.925 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:26 smithi103 ceph-mon[32635]: pgmap v361: 129 pgs: 129 active+clean; 583 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T20:09:26.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.845+0000 7f3adac00640 1 -- 172.21.15.94:0/238361264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad4106620 msgr2=0x7f3ad410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:27.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.845+0000 7f3adac00640 1 --2- 172.21.15.94:0/238361264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad4106620 0x7f3ad410ceb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009ca0 tx=0x7f3acc02a8b0 comp rx=0 tx=0).stop 2024-04-26T20:09:27.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.845+0000 7f3adac00640 1 -- 172.21.15.94:0/238361264 shutdown_connections 2024-04-26T20:09:27.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.845+0000 7f3adac00640 1 --2- 172.21.15.94:0/238361264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad4106620 0x7f3ad410ceb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:27.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.845+0000 7f3adac00640 1 --2- 172.21.15.94:0/238361264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad4105c60 0x7f3ad41060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:27.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.845+0000 7f3adac00640 1 --2- 172.21.15.94:0/238361264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ad4104a70 0x7f3ad4104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:27.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.845+0000 7f3adac00640 1 -- 172.21.15.94:0/238361264 >> 172.21.15.94:0/238361264 conn(0x7f3ad40fffe0 msgr2=0x7f3ad4102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:27.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.845+0000 7f3adac00640 1 -- 172.21.15.94:0/238361264 shutdown_connections 2024-04-26T20:09:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.845+0000 7f3adac00640 1 -- 172.21.15.94:0/238361264 wait complete. 2024-04-26T20:09:27.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.847+0000 7f3adac00640 1 Processor -- start 2024-04-26T20:09:27.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.847+0000 7f3adac00640 1 -- start start 2024-04-26T20:09:27.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.847+0000 7f3adac00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad4104a70 0x7f3ad419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:27.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.847+0000 7f3adac00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ad4105c60 0x7f3ad419b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:27.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.847+0000 7f3adac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad4106620 0x7f3ad41a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.847+0000 7f3adac00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad41105f0 con 0x7f3ad4105c60 2024-04-26T20:09:27.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.847+0000 7f3adac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad4110470 con 0x7f3ad4106620 2024-04-26T20:09:27.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.847+0000 7f3adac00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad4110770 con 0x7f3ad4104a70 2024-04-26T20:09:27.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ada200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad4106620 0x7f3ad41a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ad8e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ad4105c60 0x7f3ad419b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:27.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ada200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad4106620 0x7f3ad41a2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:50064/0 (socket says 172.21.15.94:50064) 2024-04-26T20:09:27.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ada200640 1 -- 172.21.15.94:0/2469398847 learned_addr learned my addr 172.21.15.94:0/2469398847 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:27.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ad9800640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad4104a70 0x7f3ad419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:27.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ad8e00640 1 -- 172.21.15.94:0/2469398847 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad4104a70 msgr2=0x7f3ad419abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:27.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ad8e00640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad4104a70 0x7f3ad419abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ad8e00640 1 -- 172.21.15.94:0/2469398847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad4106620 msgr2=0x7f3ad41a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:27.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ad8e00640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad4106620 0x7f3ad41a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:27.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ad8e00640 1 -- 172.21.15.94:0/2469398847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad41a27f0 con 0x7f3ad4105c60 2024-04-26T20:09:27.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.848+0000 7f3ada200640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad4106620 0x7f3ad41a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:09:27.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.849+0000 7f3ad8e00640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ad4105c60 0x7f3ad419b110 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000e9f0 tx=0x7f3ac000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:27.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.849+0000 7f3ad2000640 1 -- 172.21.15.94:0/2469398847 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000ccc0 con 0x7f3ad4105c60 2024-04-26T20:09:28.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.849+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad41a2ae0 con 0x7f3ad4105c60 2024-04-26T20:09:28.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.849+0000 7f3ad2000640 1 -- 172.21.15.94:0/2469398847 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ac0005020 con 0x7f3ad4105c60 2024-04-26T20:09:28.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.849+0000 7f3ad2000640 1 -- 172.21.15.94:0/2469398847 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0004a00 con 0x7f3ad4105c60 2024-04-26T20:09:28.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.849+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad41a2fe0 con 0x7f3ad4105c60 2024-04-26T20:09:28.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.850+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ad410bd90 con 0x7f3ad4105c60 2024-04-26T20:09:28.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.852+0000 7f3ad2000640 1 -- 172.21.15.94:0/2469398847 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3ac0004ba0 con 0x7f3ad4105c60 2024-04-26T20:09:28.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.854+0000 7f3ad2000640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3abc0a3bb0 0x7f3abc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:28.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.854+0000 7f3ad2000640 1 -- 172.21.15.94:0/2469398847 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f3ac00d0740 con 0x7f3ad4105c60 2024-04-26T20:09:28.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.854+0000 7f3ad9800640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3abc0a3bb0 0x7f3abc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:28.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.855+0000 7f3ad9800640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3abc0a3bb0 0x7f3abc0a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4105ac0 tx=0x7f3ac8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:28.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:26.858+0000 7f3ad2000640 1 -- 172.21.15.94:0/2469398847 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3ac009cd60 con 0x7f3ad4105c60 2024-04-26T20:09:28.343 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:27 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2469398847' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T20:09:28.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:27 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2469398847' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T20:09:28.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.101+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f3ad402dad0 con 0x7f3ad4105c60 2024-04-26T20:09:28.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.370+0000 7f3ad2000640 1 -- 172.21.15.94:0/2469398847 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v24) ==== 115+0+0 (secure 0 0 0) 0x7f3ac009c660 con 0x7f3ad4105c60 2024-04-26T20:09:28.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3abc0a3bb0 msgr2=0x7f3abc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:28.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3abc0a3bb0 0x7f3abc0a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4105ac0 tx=0x7f3ac8005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:28.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ad4105c60 msgr2=0x7f3ad419b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:28.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ad4105c60 0x7f3ad419b110 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000e9f0 tx=0x7f3ac000eeb0 comp rx=0 tx=0).stop 2024-04-26T20:09:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 shutdown_connections 2024-04-26T20:09:28.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3abc0a3bb0 0x7f3abc0a6070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:28.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad4106620 0x7f3ad41a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:28.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ad4105c60 0x7f3ad419b110 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:28.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 --2- 172.21.15.94:0/2469398847 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ad4104a70 0x7f3ad419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:28.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 >> 172.21.15.94:0/2469398847 conn(0x7f3ad40fffe0 msgr2=0x7f3ad4108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 shutdown_connections 2024-04-26T20:09:28.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:27.374+0000 7f3adac00640 1 -- 172.21.15.94:0/2469398847 wait complete. 2024-04-26T20:09:28.691 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-26T20:09:28.701 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-26T20:09:28.855 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:27 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2469398847' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T20:09:28.946 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:28 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2469398847' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T20:09:29.029 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:28 smithi094 ceph-mon[28050]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T20:09:29.040 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:28 smithi094 ceph-mon[28050]: pgmap v362: 129 pgs: 129 active+clean; 583 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-26T20:09:29.052 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:28 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2469398847' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T20:09:29.128 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:28 smithi103 ceph-mon[32635]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T20:09:29.164 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:28 smithi103 ceph-mon[32635]: pgmap v362: 129 pgs: 129 active+clean; 583 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-26T20:09:29.175 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:28 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2469398847' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T20:09:29.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:28 smithi204 ceph-mon[32618]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T20:09:29.218 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:28 smithi204 ceph-mon[32618]: pgmap v362: 129 pgs: 129 active+clean; 583 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-26T20:09:29.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.265+0000 7f68a6200640 1 -- 172.21.15.94:0/2735265052 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68a010a180 msgr2=0x7f68a010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:29.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.265+0000 7f68a6200640 1 --2- 172.21.15.94:0/2735265052 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68a010a180 0x7f68a010a600 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f688c009860 tx=0x7f688c02a340 comp rx=0 tx=0).stop 2024-04-26T20:09:29.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.265+0000 7f68a6200640 1 -- 172.21.15.94:0/2735265052 shutdown_connections 2024-04-26T20:09:29.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.265+0000 7f68a6200640 1 --2- 172.21.15.94:0/2735265052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68a010ab40 0x7f68a01113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:29.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.265+0000 7f68a6200640 1 --2- 172.21.15.94:0/2735265052 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68a010a180 0x7f68a010a600 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:29.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.265+0000 7f68a6200640 1 --2- 172.21.15.94:0/2735265052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68a0108f90 0x7f68a01093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:29.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.265+0000 7f68a6200640 1 -- 172.21.15.94:0/2735265052 >> 172.21.15.94:0/2735265052 conn(0x7f68a0077f30 msgr2=0x7f68a007a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:29.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.265+0000 7f68a6200640 1 -- 172.21.15.94:0/2735265052 shutdown_connections 2024-04-26T20:09:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.266+0000 7f68a6200640 1 -- 172.21.15.94:0/2735265052 wait complete. 2024-04-26T20:09:29.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.267+0000 7f68a6200640 1 Processor -- start 2024-04-26T20:09:29.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.267+0000 7f68a6200640 1 -- start start 2024-04-26T20:09:29.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.267+0000 7f68a6200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68a0108f90 0x7f68a019f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:29.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.267+0000 7f68a6200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68a010a180 0x7f68a019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:29.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.267+0000 7f68a6200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68a010ab40 0x7f68a01a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:29.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.267+0000 7f68a6200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f68a0114ad0 con 0x7f68a0108f90 2024-04-26T20:09:29.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.267+0000 7f68a6200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68a0114950 con 0x7f68a010a180 2024-04-26T20:09:29.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.267+0000 7f68a6200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f68a0114c50 con 0x7f68a010ab40 2024-04-26T20:09:29.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689f400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68a0108f90 0x7f68a019f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:29.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68a010a180 0x7f68a019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689f400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68a0108f90 0x7f68a019f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33782/0 (socket says 172.21.15.94:33782) 2024-04-26T20:09:29.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68a010a180 0x7f68a019f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:50090/0 (socket says 172.21.15.94:50090) 2024-04-26T20:09:29.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689ea00640 1 -- 172.21.15.94:0/2945579598 learned_addr learned my addr 172.21.15.94:0/2945579598 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:29.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68a010ab40 0x7f68a01a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:29.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689ea00640 1 -- 172.21.15.94:0/2945579598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68a010ab40 msgr2=0x7f68a01a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:29.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689ea00640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68a010ab40 0x7f68a01a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:29.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689ea00640 1 -- 172.21.15.94:0/2945579598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68a0108f90 msgr2=0x7f68a019f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:29.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689ea00640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68a0108f90 0x7f68a019f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:29.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689ea00640 1 -- 172.21.15.94:0/2945579598 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a01a6e00 con 0x7f68a010a180 2024-04-26T20:09:29.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.268+0000 7f689ea00640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68a010a180 0x7f68a019f720 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f688c005940 tx=0x7f688c02ae20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:29.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.269+0000 7f6897e00640 1 -- 172.21.15.94:0/2945579598 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c040020 con 0x7f68a010a180 2024-04-26T20:09:29.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.269+0000 7f6897e00640 1 -- 172.21.15.94:0/2945579598 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f688c030e50 con 0x7f68a010a180 2024-04-26T20:09:30.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.269+0000 7f6897e00640 1 -- 172.21.15.94:0/2945579598 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c037040 con 0x7f68a010a180 2024-04-26T20:09:30.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.269+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a01a7090 con 0x7f68a010a180 2024-04-26T20:09:30.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.269+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a01a7530 con 0x7f68a010a180 2024-04-26T20:09:30.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.270+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a0110380 con 0x7f68a010a180 2024-04-26T20:09:30.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.272+0000 7f6897e00640 1 -- 172.21.15.94:0/2945579598 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f688c043020 con 0x7f68a010a180 2024-04-26T20:09:30.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.273+0000 7f6897e00640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68840a3c30 0x7f68840a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:30.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.274+0000 7f6897e00640 1 -- 172.21.15.94:0/2945579598 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f688c0f0d20 con 0x7f68a010a180 2024-04-26T20:09:30.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.274+0000 7f689f400640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68840a3c30 0x7f68840a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:30.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.274+0000 7f689f400640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68840a3c30 0x7f68840a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f68a0109fe0 tx=0x7f6890005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:30.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.276+0000 7f6897e00640 1 -- 172.21.15.94:0/2945579598 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f688c04a020 con 0x7f68a010a180 2024-04-26T20:09:30.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.505+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f68a0062d70 con 0x7f68840a3c30 2024-04-26T20:09:30.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.544+0000 7f6897e00640 1 -- 172.21.15.94:0/2945579598 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f68a0062d70 con 0x7f68840a3c30 2024-04-26T20:09:30.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68840a3c30 msgr2=0x7f68840a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:30.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68840a3c30 0x7f68840a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f68a0109fe0 tx=0x7f6890005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:30.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68a010a180 msgr2=0x7f68a019f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:30.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68a010a180 0x7f68a019f720 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f688c005940 tx=0x7f688c02ae20 comp rx=0 tx=0).stop 2024-04-26T20:09:30.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 shutdown_connections 2024-04-26T20:09:30.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68840a3c30 0x7f68840a60f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68a010ab40 0x7f68a01a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:30.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68a010a180 0x7f68a019f720 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:30.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 --2- 172.21.15.94:0/2945579598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68a0108f90 0x7f68a019f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:30.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 >> 172.21.15.94:0/2945579598 conn(0x7f68a0077f30 msgr2=0x7f68a010d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:30.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 shutdown_connections 2024-04-26T20:09:30.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:29.547+0000 7f68a6200640 1 -- 172.21.15.94:0/2945579598 wait complete. 2024-04-26T20:09:30.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T20:09:30.664 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:09:29 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:09:29.519+0000 7f0273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:09:30.672 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:09:29 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:09:29.519+0000 7f0273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:09:30.680 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:09:29 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:09:29.519+0000 7f0273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:09:30.688 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:09:29 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:09:29.519+0000 7f0273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:09:30.696 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:09:29 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:09:29.519+0000 7f0273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:09:30.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.908+0000 7f3dba200640 1 -- 172.21.15.94:0/3742277747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3db4106620 msgr2=0x7f3db410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:30.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.908+0000 7f3dba200640 1 --2- 172.21.15.94:0/3742277747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3db4106620 0x7f3db410ceb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009be0 tx=0x7f3dac02a6c0 comp rx=0 tx=0).stop 2024-04-26T20:09:30.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.908+0000 7f3dba200640 1 -- 172.21.15.94:0/3742277747 shutdown_connections 2024-04-26T20:09:30.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.908+0000 7f3dba200640 1 --2- 172.21.15.94:0/3742277747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3db4106620 0x7f3db410ceb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:30.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.909+0000 7f3dba200640 1 --2- 172.21.15.94:0/3742277747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3db4105c60 0x7f3db41060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:30.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.909+0000 7f3dba200640 1 --2- 172.21.15.94:0/3742277747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3db4104a70 0x7f3db4104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:31.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.909+0000 7f3dba200640 1 -- 172.21.15.94:0/3742277747 >> 172.21.15.94:0/3742277747 conn(0x7f3db40fffe0 msgr2=0x7f3db4102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:31.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.909+0000 7f3dba200640 1 -- 172.21.15.94:0/3742277747 shutdown_connections 2024-04-26T20:09:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.909+0000 7f3dba200640 1 -- 172.21.15.94:0/3742277747 wait complete. 2024-04-26T20:09:31.120 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:30 smithi103 ceph-mon[32635]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:31.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:30 smithi103 ceph-mon[32635]: pgmap v363: 129 pgs: 129 active+clean; 584 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-26T20:09:31.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.910+0000 7f3dba200640 1 Processor -- start 2024-04-26T20:09:31.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.910+0000 7f3dba200640 1 -- start start 2024-04-26T20:09:31.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.910+0000 7f3dba200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3db4104a70 0x7f3db407a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.910+0000 7f3dba200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3db4105c60 0x7f3db407ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.910+0000 7f3dba200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3db4106620 0x7f3db4077250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.910+0000 7f3dba200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3db4110580 con 0x7f3db4105c60 2024-04-26T20:09:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.910+0000 7f3dba200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3db4110400 con 0x7f3db4106620 2024-04-26T20:09:31.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.910+0000 7f3dba200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3db4110700 con 0x7f3db4104a70 2024-04-26T20:09:31.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3db4105c60 0x7f3db407ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:31.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3db4104a70 0x7f3db407a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:31.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3db4105c60 0x7f3db407ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55014/0 (socket says 172.21.15.94:55014) 2024-04-26T20:09:31.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db3e00640 1 -- 172.21.15.94:0/3073388862 learned_addr learned my addr 172.21.15.94:0/3073388862 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:31.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db9800640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3db4106620 0x7f3db4077250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:31.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db3e00640 1 -- 172.21.15.94:0/3073388862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3db4104a70 msgr2=0x7f3db407a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:31.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db3e00640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3db4104a70 0x7f3db407a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:31.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db3e00640 1 -- 172.21.15.94:0/3073388862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3db4106620 msgr2=0x7f3db4077250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:31.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db3e00640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3db4106620 0x7f3db4077250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:31.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db3e00640 1 -- 172.21.15.94:0/3073388862 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db4077b10 con 0x7f3db4105c60 2024-04-26T20:09:31.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.911+0000 7f3db3e00640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3db4105c60 0x7f3db407ac20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3da800e9c0 tx=0x7f3da800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:31.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.912+0000 7f3db1600640 1 -- 172.21.15.94:0/3073388862 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da800ccc0 con 0x7f3db4105c60 2024-04-26T20:09:31.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.912+0000 7f3db1600640 1 -- 172.21.15.94:0/3073388862 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3da80054d0 con 0x7f3db4105c60 2024-04-26T20:09:31.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.912+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db41a7050 con 0x7f3db4105c60 2024-04-26T20:09:31.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.912+0000 7f3db1600640 1 -- 172.21.15.94:0/3073388862 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8010430 con 0x7f3db4105c60 2024-04-26T20:09:31.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.912+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db41a7460 con 0x7f3db4105c60 2024-04-26T20:09:31.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.913+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db410bd90 con 0x7f3db4105c60 2024-04-26T20:09:31.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.916+0000 7f3db1600640 1 -- 172.21.15.94:0/3073388862 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3da8004f80 con 0x7f3db4105c60 2024-04-26T20:09:31.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.917+0000 7f3db1600640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d9c0a3c40 0x7f3d9c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:31.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.917+0000 7f3db1600640 1 -- 172.21.15.94:0/3073388862 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f3da80d0d70 con 0x7f3db4105c60 2024-04-26T20:09:31.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.918+0000 7f3db8e00640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d9c0a3c40 0x7f3d9c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:31.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.918+0000 7f3db8e00640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d9c0a3c40 0x7f3d9c0a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3db4105ac0 tx=0x7f3da4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:31.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:30.922+0000 7f3db1600640 1 -- 172.21.15.94:0/3073388862 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3da809d230 con 0x7f3db4105c60 2024-04-26T20:09:31.754 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:30 smithi094 ceph-mon[28050]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:31.762 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:30 smithi094 ceph-mon[28050]: pgmap v363: 129 pgs: 129 active+clean; 584 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-26T20:09:31.849 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:30 smithi204 ceph-mon[32618]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:31.860 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:30 smithi204 ceph-mon[32618]: pgmap v363: 129 pgs: 129 active+clean; 584 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-26T20:09:31.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.158+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3db4062d70 con 0x7f3d9c0a3c40 2024-04-26T20:09:31.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.196+0000 7f3db1600640 1 -- 172.21.15.94:0/3073388862 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3db4062d70 con 0x7f3d9c0a3c40 2024-04-26T20:09:31.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d9c0a3c40 msgr2=0x7f3d9c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d9c0a3c40 0x7f3d9c0a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3db4105ac0 tx=0x7f3da4005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:31.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3db4105c60 msgr2=0x7f3db407ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:31.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3db4105c60 0x7f3db407ac20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3da800e9c0 tx=0x7f3da800ee80 comp rx=0 tx=0).stop 2024-04-26T20:09:31.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 shutdown_connections 2024-04-26T20:09:31.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d9c0a3c40 0x7f3d9c0a6100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:31.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3db4106620 0x7f3db4077250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:31.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3db4105c60 0x7f3db407ac20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:31.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 --2- 172.21.15.94:0/3073388862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3db4104a70 0x7f3db407a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 >> 172.21.15.94:0/3073388862 conn(0x7f3db40fffe0 msgr2=0x7f3db4108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:32.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 shutdown_connections 2024-04-26T20:09:32.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:31.200+0000 7f3dba200640 1 -- 172.21.15.94:0/3073388862 wait complete. 2024-04-26T20:09:32.103 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T20:09:32.161 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:31 smithi094 ceph-mon[28050]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-04-26T20:09:32.170 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:31 smithi103 ceph-mon[32635]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-04-26T20:09:32.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:31 smithi204 ceph-mon[32618]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-04-26T20:09:32.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.386+0000 7f050d200640 1 -- 172.21.15.94:0/1040808122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0508109fc0 msgr2=0x7f050810a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:32.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.386+0000 7f050d200640 1 --2- 172.21.15.94:0/1040808122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0508109fc0 0x7f050810a440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009900 tx=0x7f04f002a410 comp rx=0 tx=0).stop 2024-04-26T20:09:32.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.387+0000 7f050d200640 1 -- 172.21.15.94:0/1040808122 shutdown_connections 2024-04-26T20:09:32.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.387+0000 7f050d200640 1 --2- 172.21.15.94:0/1040808122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f050810a980 0x7f05081112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:32.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.387+0000 7f050d200640 1 --2- 172.21.15.94:0/1040808122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0508109fc0 0x7f050810a440 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:32.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.387+0000 7f050d200640 1 --2- 172.21.15.94:0/1040808122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0508108e70 0x7f0508109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:32.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.387+0000 7f050d200640 1 -- 172.21.15.94:0/1040808122 >> 172.21.15.94:0/1040808122 conn(0x7f0508077e00 msgr2=0x7f050807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:32.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.387+0000 7f050d200640 1 -- 172.21.15.94:0/1040808122 shutdown_connections 2024-04-26T20:09:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.387+0000 7f050d200640 1 -- 172.21.15.94:0/1040808122 wait complete. 2024-04-26T20:09:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.388+0000 7f050d200640 1 Processor -- start 2024-04-26T20:09:32.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.388+0000 7f050d200640 1 -- start start 2024-04-26T20:09:32.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.388+0000 7f050d200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0508108e70 0x7f050819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.388+0000 7f050d200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0508109fc0 0x7f050819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:32.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.388+0000 7f050d200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f050810a980 0x7f05081a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:32.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.388+0000 7f050d200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f05081149a0 con 0x7f050810a980 2024-04-26T20:09:32.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.388+0000 7f050d200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0508114820 con 0x7f0508108e70 2024-04-26T20:09:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.388+0000 7f050d200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0508114b20 con 0x7f0508109fc0 2024-04-26T20:09:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0508108e70 0x7f050819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:32.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0508108e70 0x7f050819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:56318/0 (socket says 172.21.15.94:56318) 2024-04-26T20:09:32.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507400640 1 -- 172.21.15.94:0/84144538 learned_addr learned my addr 172.21.15.94:0/84144538 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:32.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507e00640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f050810a980 0x7f05081a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:32.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f04ffe00640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0508109fc0 0x7f050819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:32.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507400640 1 -- 172.21.15.94:0/84144538 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0508109fc0 msgr2=0x7f050819f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:32.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507400640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0508109fc0 0x7f050819f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:32.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507400640 1 -- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f050810a980 msgr2=0x7f05081a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:32.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507400640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f050810a980 0x7f05081a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:32.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507400640 1 -- 172.21.15.94:0/84144538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05081a6c60 con 0x7f0508108e70 2024-04-26T20:09:33.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.389+0000 7f0507400640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0508108e70 0x7f050819f040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f04f800e9f0 tx=0x7f04f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:33.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.390+0000 7f0504c00640 1 -- 172.21.15.94:0/84144538 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f800cd80 con 0x7f0508108e70 2024-04-26T20:09:33.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.390+0000 7f0504c00640 1 -- 172.21.15.94:0/84144538 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04f80108e0 con 0x7f0508108e70 2024-04-26T20:09:33.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.390+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05081a6f50 con 0x7f0508108e70 2024-04-26T20:09:33.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.390+0000 7f0504c00640 1 -- 172.21.15.94:0/84144538 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f8018980 con 0x7f0508108e70 2024-04-26T20:09:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.390+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05081a73b0 con 0x7f0508108e70 2024-04-26T20:09:33.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.391+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0508110200 con 0x7f0508108e70 2024-04-26T20:09:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.393+0000 7f0504c00640 1 -- 172.21.15.94:0/84144538 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f04f8010430 con 0x7f0508108e70 2024-04-26T20:09:33.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.394+0000 7f0504c00640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04ec0a3c40 0x7f04ec0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:33.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.394+0000 7f0504c00640 1 -- 172.21.15.94:0/84144538 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f04f80d0cb0 con 0x7f0508108e70 2024-04-26T20:09:33.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.395+0000 7f04ffe00640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04ec0a3c40 0x7f04ec0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:33.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.395+0000 7f04ffe00640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04ec0a3c40 0x7f04ec0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f04f0000c00 tx=0x7f04f00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:33.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.399+0000 7f0504c00640 1 -- 172.21.15.94:0/84144538 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f04f809d170 con 0x7f0508108e70 2024-04-26T20:09:33.268 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:32 smithi204 ceph-mon[32618]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:33.277 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:32 smithi204 ceph-mon[32618]: pgmap v364: 129 pgs: 129 active+clean; 590 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-04-26T20:09:33.287 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:32 smithi103 ceph-mon[32635]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:33.296 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:32 smithi103 ceph-mon[32635]: pgmap v364: 129 pgs: 129 active+clean; 590 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-04-26T20:09:33.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.629+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f0508062e00 con 0x7f04ec0a3c40 2024-04-26T20:09:33.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.644+0000 7f0504c00640 1 -- 172.21.15.94:0/84144538 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0508062e00 con 0x7f04ec0a3c40 2024-04-26T20:09:33.378 INFO:teuthology.orchestra.run.smithi094.stdout:/volumes/qa/sv_0/42cfdad6-e856-4b26-a59d-ed473a13226a 2024-04-26T20:09:33.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04ec0a3c40 msgr2=0x7f04ec0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:33.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04ec0a3c40 0x7f04ec0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f04f0000c00 tx=0x7f04f00314a0 comp rx=0 tx=0).stop 2024-04-26T20:09:33.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0508108e70 msgr2=0x7f050819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:33.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0508108e70 0x7f050819f040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f04f800e9f0 tx=0x7f04f800eeb0 comp rx=0 tx=0).stop 2024-04-26T20:09:33.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 shutdown_connections 2024-04-26T20:09:33.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f04ec0a3c40 0x7f04ec0a6100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:33.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f050810a980 0x7f05081a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0508109fc0 0x7f050819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:33.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 --2- 172.21.15.94:0/84144538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0508108e70 0x7f050819f040 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:33.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 >> 172.21.15.94:0/84144538 conn(0x7f0508077e00 msgr2=0x7f050810d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:33.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 shutdown_connections 2024-04-26T20:09:33.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:32.647+0000 7f050d200640 1 -- 172.21.15.94:0/84144538 wait complete. 2024-04-26T20:09:33.667 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T20:09:33.675 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:32 smithi094 ceph-mon[28050]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:33.708 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:32 smithi094 ceph-mon[28050]: pgmap v364: 129 pgs: 129 active+clean; 590 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-04-26T20:09:33.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.934+0000 7f991aa00640 1 -- 172.21.15.94:0/937734802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9914105c60 msgr2=0x7f99141060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:33.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.934+0000 7f991aa00640 1 --2- 172.21.15.94:0/937734802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9914105c60 0x7f99141060e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9900009900 tx=0x7f990002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:09:34.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.935+0000 7f991aa00640 1 -- 172.21.15.94:0/937734802 shutdown_connections 2024-04-26T20:09:34.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.935+0000 7f991aa00640 1 --2- 172.21.15.94:0/937734802 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9914106620 0x7f991410ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:34.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.935+0000 7f991aa00640 1 --2- 172.21.15.94:0/937734802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9914105c60 0x7f99141060e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:34.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.935+0000 7f991aa00640 1 --2- 172.21.15.94:0/937734802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9914104a70 0x7f9914104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:34.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.935+0000 7f991aa00640 1 -- 172.21.15.94:0/937734802 >> 172.21.15.94:0/937734802 conn(0x7f99140fffe0 msgr2=0x7f9914102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:34.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.935+0000 7f991aa00640 1 -- 172.21.15.94:0/937734802 shutdown_connections 2024-04-26T20:09:34.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.935+0000 7f991aa00640 1 -- 172.21.15.94:0/937734802 wait complete. 2024-04-26T20:09:34.112 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:33 smithi103 ceph-mon[32635]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.936+0000 7f991aa00640 1 Processor -- start 2024-04-26T20:09:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.936+0000 7f991aa00640 1 -- start start 2024-04-26T20:09:34.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.936+0000 7f991aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9914104a70 0x7f991419ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:34.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.936+0000 7f991aa00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9914105c60 0x7f991419b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.936+0000 7f991aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9914106620 0x7f99141a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:34.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.936+0000 7f991aa00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f99141105f0 con 0x7f9914105c60 2024-04-26T20:09:34.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.936+0000 7f991aa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9914110470 con 0x7f9914104a70 2024-04-26T20:09:34.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.936+0000 7f991aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9914110770 con 0x7f9914106620 2024-04-26T20:09:34.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.936+0000 7f9918c00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9914105c60 0x7f991419b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:34.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9918c00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9914105c60 0x7f991419b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55060/0 (socket says 172.21.15.94:55060) 2024-04-26T20:09:34.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9918c00640 1 -- 172.21.15.94:0/2954586388 learned_addr learned my addr 172.21.15.94:0/2954586388 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:34.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9919600640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9914104a70 0x7f991419ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f991a000640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9914106620 0x7f99141a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:34.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9918c00640 1 -- 172.21.15.94:0/2954586388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9914106620 msgr2=0x7f99141a2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:34.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9918c00640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9914106620 0x7f99141a2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:34.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9918c00640 1 -- 172.21.15.94:0/2954586388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9914104a70 msgr2=0x7f991419ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:34.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9918c00640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9914104a70 0x7f991419ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:34.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9918c00640 1 -- 172.21.15.94:0/2954586388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99141a2920 con 0x7f9914105c60 2024-04-26T20:09:34.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9918c00640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9914105c60 0x7f991419b250 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9900005ab0 tx=0x7f9900003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:34.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.937+0000 7f9912000640 1 -- 172.21.15.94:0/2954586388 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990003f070 con 0x7f9914105c60 2024-04-26T20:09:34.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.938+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99141a2bb0 con 0x7f9914105c60 2024-04-26T20:09:34.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.938+0000 7f9912000640 1 -- 172.21.15.94:0/2954586388 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9900031c50 con 0x7f9914105c60 2024-04-26T20:09:34.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.938+0000 7f9912000640 1 -- 172.21.15.94:0/2954586388 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9900035030 con 0x7f9914105c60 2024-04-26T20:09:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.938+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99141a30f0 con 0x7f9914105c60 2024-04-26T20:09:34.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.939+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f991410be00 con 0x7f9914105c60 2024-04-26T20:09:34.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.941+0000 7f9912000640 1 -- 172.21.15.94:0/2954586388 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9900042070 con 0x7f9914105c60 2024-04-26T20:09:34.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.942+0000 7f9912000640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f98fc0a3c40 0x7f98fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.942+0000 7f9912000640 1 -- 172.21.15.94:0/2954586388 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f99000f1070 con 0x7f9914105c60 2024-04-26T20:09:34.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.943+0000 7f9919600640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f98fc0a3c40 0x7f98fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:34.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.943+0000 7f9919600640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f98fc0a3c40 0x7f98fc0a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f991419c1e0 tx=0x7f9908005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:34.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:33.947+0000 7f9912000640 1 -- 172.21.15.94:0/2954586388 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f99000bd530 con 0x7f9914105c60 2024-04-26T20:09:34.752 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:33 smithi094 ceph-mon[28050]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:34.759 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:33 smithi204 ceph-mon[32618]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:34.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.180+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9914062d70 con 0x7f98fc0a3c40 2024-04-26T20:09:34.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.220+0000 7f9912000640 1 -- 172.21.15.94:0/2954586388 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9914062d70 con 0x7f98fc0a3c40 2024-04-26T20:09:34.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f98fc0a3c40 msgr2=0x7f98fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:34.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f98fc0a3c40 0x7f98fc0a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f991419c1e0 tx=0x7f9908005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9914105c60 msgr2=0x7f991419b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:34.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9914105c60 0x7f991419b250 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9900005ab0 tx=0x7f9900003b90 comp rx=0 tx=0).stop 2024-04-26T20:09:34.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 shutdown_connections 2024-04-26T20:09:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f98fc0a3c40 0x7f98fc0a6100 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:34.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9914106620 0x7f99141a2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:34.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9914105c60 0x7f991419b250 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:35.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 --2- 172.21.15.94:0/2954586388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9914104a70 0x7f991419ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 >> 172.21.15.94:0/2954586388 conn(0x7f99140fffe0 msgr2=0x7f9914108fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:35.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.223+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 shutdown_connections 2024-04-26T20:09:35.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:34.224+0000 7f991aa00640 1 -- 172.21.15.94:0/2954586388 wait complete. 2024-04-26T20:09:35.070 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T20:09:35.077 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:34 smithi094 ceph-mon[28050]: pgmap v365: 129 pgs: 129 active+clean; 591 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.1 KiB/s wr, 13 op/s 2024-04-26T20:09:35.084 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:34 smithi103 ceph-mon[32635]: pgmap v365: 129 pgs: 129 active+clean; 591 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.1 KiB/s wr, 13 op/s 2024-04-26T20:09:35.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:34 smithi204 ceph-mon[32618]: pgmap v365: 129 pgs: 129 active+clean; 591 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.1 KiB/s wr, 13 op/s 2024-04-26T20:09:35.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.302+0000 7f4e43e00640 1 -- 172.21.15.94:0/3077248570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 msgr2=0x7f4e4410a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:35.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.302+0000 7f4e43e00640 1 --2- 172.21.15.94:0/3077248570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 0x7f4e4410a440 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4e30009900 tx=0x7f4e3002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:09:35.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.302+0000 7f4e43e00640 1 -- 172.21.15.94:0/3077248570 shutdown_connections 2024-04-26T20:09:35.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.302+0000 7f4e43e00640 1 --2- 172.21.15.94:0/3077248570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e4410a980 0x7f4e441112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:35.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.302+0000 7f4e43e00640 1 --2- 172.21.15.94:0/3077248570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 0x7f4e4410a440 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:35.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.302+0000 7f4e43e00640 1 --2- 172.21.15.94:0/3077248570 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e44108e70 0x7f4e44109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:35.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.302+0000 7f4e43e00640 1 -- 172.21.15.94:0/3077248570 >> 172.21.15.94:0/3077248570 conn(0x7f4e44077e00 msgr2=0x7f4e4407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:35.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.302+0000 7f4e43e00640 1 -- 172.21.15.94:0/3077248570 shutdown_connections 2024-04-26T20:09:35.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.302+0000 7f4e43e00640 1 -- 172.21.15.94:0/3077248570 wait complete. 2024-04-26T20:09:35.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.303+0000 7f4e43e00640 1 Processor -- start 2024-04-26T20:09:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.303+0000 7f4e43e00640 1 -- start start 2024-04-26T20:09:35.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.303+0000 7f4e43e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e44108e70 0x7f4e4419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:35.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.303+0000 7f4e43e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 0x7f4e4419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:35.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.303+0000 7f4e43e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e4410a980 0x7f4e441a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:35.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.303+0000 7f4e43e00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4e441149c0 con 0x7f4e44109fc0 2024-04-26T20:09:35.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.303+0000 7f4e43e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e44114840 con 0x7f4e4410a980 2024-04-26T20:09:35.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e43e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4e44114b40 con 0x7f4e44108e70 2024-04-26T20:09:35.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e43400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e4410a980 0x7f4e441a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:35.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e42000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 0x7f4e4419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:35.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e43400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e4410a980 0x7f4e441a6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:56360/0 (socket says 172.21.15.94:56360) 2024-04-26T20:09:35.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e42000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 0x7f4e4419f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55070/0 (socket says 172.21.15.94:55070) 2024-04-26T20:09:35.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e43400640 1 -- 172.21.15.94:0/2548277053 learned_addr learned my addr 172.21.15.94:0/2548277053 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:35.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e42a00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e44108e70 0x7f4e4419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:35.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e43400640 1 -- 172.21.15.94:0/2548277053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e44108e70 msgr2=0x7f4e4419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:35.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e43400640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e44108e70 0x7f4e4419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e43400640 1 -- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 msgr2=0x7f4e4419f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:35.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e43400640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 0x7f4e4419f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:36.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e43400640 1 -- 172.21.15.94:0/2548277053 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e441a6cb0 con 0x7f4e4410a980 2024-04-26T20:09:36.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.304+0000 7f4e42000640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 0x7f4e4419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:09:36.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.305+0000 7f4e43400640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e4410a980 0x7f4e441a6640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00eb10 tx=0x7f4e3c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.305+0000 7f4e48a00640 1 -- 172.21.15.94:0/2548277053 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c004b20 con 0x7f4e4410a980 2024-04-26T20:09:36.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.305+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e441a6fa0 con 0x7f4e4410a980 2024-04-26T20:09:36.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.305+0000 7f4e48a00640 1 -- 172.21.15.94:0/2548277053 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e3c004cc0 con 0x7f4e4410a980 2024-04-26T20:09:36.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.305+0000 7f4e48a00640 1 -- 172.21.15.94:0/2548277053 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c010670 con 0x7f4e4410a980 2024-04-26T20:09:36.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.305+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e441a7410 con 0x7f4e4410a980 2024-04-26T20:09:36.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.306+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e44110250 con 0x7f4e4410a980 2024-04-26T20:09:36.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.308+0000 7f4e48a00640 1 -- 172.21.15.94:0/2548277053 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4e3c005020 con 0x7f4e4410a980 2024-04-26T20:09:36.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.309+0000 7f4e48a00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4e280a3c40 0x7f4e280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.310+0000 7f4e48a00640 1 -- 172.21.15.94:0/2548277053 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f4e3c0d0fe0 con 0x7f4e4410a980 2024-04-26T20:09:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.310+0000 7f4e42a00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4e280a3c40 0x7f4e280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:36.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.310+0000 7f4e42a00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4e280a3c40 0x7f4e280a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009bd0 tx=0x7f4e34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:36.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.312+0000 7f4e48a00640 1 -- 172.21.15.94:0/2548277053 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4e3c09d4a0 con 0x7f4e4410a980 2024-04-26T20:09:36.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.550+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f4e44062d70 con 0x7f4e280a3c40 2024-04-26T20:09:36.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.558+0000 7f4e48a00640 1 -- 172.21.15.94:0/2548277053 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4e44062d70 con 0x7f4e280a3c40 2024-04-26T20:09:36.304 INFO:teuthology.orchestra.run.smithi094.stdout:/volumes/qa/sv_1/00f14d13-c2d7-4e46-a7d3-86627f7ad110 2024-04-26T20:09:36.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4e280a3c40 msgr2=0x7f4e280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:36.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4e280a3c40 0x7f4e280a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009bd0 tx=0x7f4e34005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:36.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e4410a980 msgr2=0x7f4e441a6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:36.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e4410a980 0x7f4e441a6640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00eb10 tx=0x7f4e3c00c2d0 comp rx=0 tx=0).stop 2024-04-26T20:09:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 shutdown_connections 2024-04-26T20:09:36.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4e280a3c40 0x7f4e280a6100 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:36.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e4410a980 0x7f4e441a6640 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:36.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e44109fc0 0x7f4e4419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:36.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 --2- 172.21.15.94:0/2548277053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e44108e70 0x7f4e4419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:36.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.561+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 >> 172.21.15.94:0/2548277053 conn(0x7f4e44077e00 msgr2=0x7f4e4410d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:36.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.562+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 shutdown_connections 2024-04-26T20:09:36.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:35.562+0000 7f4e43e00640 1 -- 172.21.15.94:0/2548277053 wait complete. 2024-04-26T20:09:36.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:09:36.501 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:35 smithi094 ceph-mon[28050]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:36.564 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:35 smithi204 ceph-mon[32618]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:36.645 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:35 smithi103 ceph-mon[32635]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:36.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.778+0000 7f9508600640 1 -- 172.21.15.94:0/497552054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f950010a180 msgr2=0x7f950010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:36.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.778+0000 7f9508600640 1 --2- 172.21.15.94:0/497552054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f950010a180 0x7f950010a600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f94ec009860 tx=0x7f94ec02a340 comp rx=0 tx=0).stop 2024-04-26T20:09:36.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.778+0000 7f9508600640 1 -- 172.21.15.94:0/497552054 shutdown_connections 2024-04-26T20:09:36.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.778+0000 7f9508600640 1 --2- 172.21.15.94:0/497552054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f950010ab40 0x7f95001113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:36.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.778+0000 7f9508600640 1 --2- 172.21.15.94:0/497552054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f950010a180 0x7f950010a600 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:36.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.778+0000 7f9508600640 1 --2- 172.21.15.94:0/497552054 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9500108f90 0x7f95001093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:36.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.778+0000 7f9508600640 1 -- 172.21.15.94:0/497552054 >> 172.21.15.94:0/497552054 conn(0x7f9500077f40 msgr2=0x7f950007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:36.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.779+0000 7f9508600640 1 -- 172.21.15.94:0/497552054 shutdown_connections 2024-04-26T20:09:36.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.779+0000 7f9508600640 1 -- 172.21.15.94:0/497552054 wait complete. 2024-04-26T20:09:36.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.779+0000 7f9508600640 1 Processor -- start 2024-04-26T20:09:36.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9508600640 1 -- start start 2024-04-26T20:09:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9508600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9500108f90 0x7f950019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:36.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9508600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f950010a180 0x7f950019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:36.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9508600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f950010ab40 0x7f95001a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9508600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9500114ad0 con 0x7f950010a180 2024-04-26T20:09:37.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9508600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9500114950 con 0x7f950010ab40 2024-04-26T20:09:37.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9508600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9500114c50 con 0x7f9500108f90 2024-04-26T20:09:37.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9505200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f950010a180 0x7f950019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:37.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9505200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f950010a180 0x7f950019f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55102/0 (socket says 172.21.15.94:55102) 2024-04-26T20:09:37.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9505200640 1 -- 172.21.15.94:0/2777649762 learned_addr learned my addr 172.21.15.94:0/2777649762 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:37.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9506600640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f950010ab40 0x7f95001a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:37.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.780+0000 7f9505c00640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9500108f90 0x7f950019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:37.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f9505200640 1 -- 172.21.15.94:0/2777649762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9500108f90 msgr2=0x7f950019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:37.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f9505200640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9500108f90 0x7f950019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:37.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f9505200640 1 -- 172.21.15.94:0/2777649762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f950010ab40 msgr2=0x7f95001a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:37.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f9505200640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f950010ab40 0x7f95001a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:37.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f9505200640 1 -- 172.21.15.94:0/2777649762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95001a6dc0 con 0x7f950010a180 2024-04-26T20:09:37.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f9505200640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f950010a180 0x7f950019f6e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f94ec0057c0 tx=0x7f94ec032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:37.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f94fe000640 1 -- 172.21.15.94:0/2777649762 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec037040 con 0x7f950010a180 2024-04-26T20:09:37.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f94fe000640 1 -- 172.21.15.94:0/2777649762 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94ec030bd0 con 0x7f950010a180 2024-04-26T20:09:37.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95001a7050 con 0x7f950010a180 2024-04-26T20:09:37.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f94fe000640 1 -- 172.21.15.94:0/2777649762 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec005280 con 0x7f950010a180 2024-04-26T20:09:37.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.781+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95001a74e0 con 0x7f950010a180 2024-04-26T20:09:37.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.782+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9500110300 con 0x7f950010a180 2024-04-26T20:09:37.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.784+0000 7f94fe000640 1 -- 172.21.15.94:0/2777649762 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f94ec043020 con 0x7f950010a180 2024-04-26T20:09:37.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.787+0000 7f94fe000640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94e80a3c40 0x7f94e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.787+0000 7f94fe000640 1 -- 172.21.15.94:0/2777649762 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f94ec0efe50 con 0x7f950010a180 2024-04-26T20:09:37.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.787+0000 7f9505c00640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94e80a3c40 0x7f94e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:37.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.788+0000 7f9505c00640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94e80a3c40 0x7f94e80a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9500109fe0 tx=0x7f94f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:37.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:36.792+0000 7f94fe000640 1 -- 172.21.15.94:0/2777649762 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f94ec0bc420 con 0x7f950010a180 2024-04-26T20:09:37.697 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:36 smithi204 ceph-mon[32618]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:37.708 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:36 smithi204 ceph-mon[32618]: pgmap v366: 129 pgs: 129 active+clean; 623 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-04-26T20:09:37.718 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:36 smithi103 ceph-mon[32635]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:37.730 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:36 smithi103 ceph-mon[32635]: pgmap v366: 129 pgs: 129 active+clean; 623 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-04-26T20:09:37.742 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:36 smithi094 ceph-mon[28050]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:09:37.750 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:36 smithi094 ceph-mon[28050]: pgmap v366: 129 pgs: 129 active+clean; 623 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-04-26T20:09:37.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.036+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f950002dd20 con 0x7f950010a180 2024-04-26T20:09:37.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.038+0000 7f94fe000640 1 -- 172.21.15.94:0/2777649762 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11878 (secure 0 0 0) 0x7f94ec0bbd20 con 0x7f950010a180 2024-04-26T20:09:37.871 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:37.910 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:09:27.103941+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24562,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24562,24541]},"id":1}]} 2024-04-26T20:09:38.018 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-26T20:09:38.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94e80a3c40 msgr2=0x7f94e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94e80a3c40 0x7f94e80a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9500109fe0 tx=0x7f94f4009260 comp rx=0 tx=0).stop 2024-04-26T20:09:38.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f950010a180 msgr2=0x7f950019f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:38.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f950010a180 0x7f950019f6e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f94ec0057c0 tx=0x7f94ec032da0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 shutdown_connections 2024-04-26T20:09:38.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f94e80a3c40 0x7f94e80a6100 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f950010ab40 0x7f95001a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f950010a180 0x7f950019f6e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 --2- 172.21.15.94:0/2777649762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9500108f90 0x7f950019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 >> 172.21.15.94:0/2777649762 conn(0x7f9500077f40 msgr2=0x7f950010d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:38.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 shutdown_connections 2024-04-26T20:09:38.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:37.041+0000 7f9508600640 1 -- 172.21.15.94:0/2777649762 wait complete. 2024-04-26T20:09:38.128 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:09:38.166 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:37 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2777649762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:09:38.234 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:37 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2777649762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:09:38.273 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:37 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2777649762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:09:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.445+0000 7f7ea6600640 1 -- 172.21.15.94:0/1903668436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea0108f90 msgr2=0x7f7ea01093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:38.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.445+0000 7f7ea6600640 1 --2- 172.21.15.94:0/1903668436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea0108f90 0x7f7ea01093b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009900 tx=0x7f7e9002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.445+0000 7f7ea6600640 1 -- 172.21.15.94:0/1903668436 shutdown_connections 2024-04-26T20:09:38.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.445+0000 7f7ea6600640 1 --2- 172.21.15.94:0/1903668436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea010ab40 0x7f7ea01113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.445+0000 7f7ea6600640 1 --2- 172.21.15.94:0/1903668436 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ea010a180 0x7f7ea010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.445+0000 7f7ea6600640 1 --2- 172.21.15.94:0/1903668436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea0108f90 0x7f7ea01093b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.445+0000 7f7ea6600640 1 -- 172.21.15.94:0/1903668436 >> 172.21.15.94:0/1903668436 conn(0x7f7ea0077f40 msgr2=0x7f7ea007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:38.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.445+0000 7f7ea6600640 1 -- 172.21.15.94:0/1903668436 shutdown_connections 2024-04-26T20:09:38.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.445+0000 7f7ea6600640 1 -- 172.21.15.94:0/1903668436 wait complete. 2024-04-26T20:09:38.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.446+0000 7f7ea6600640 1 Processor -- start 2024-04-26T20:09:38.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.446+0000 7f7ea6600640 1 -- start start 2024-04-26T20:09:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7ea6600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ea0108f90 0x7f7ea019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:38.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7ea6600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea010a180 0x7f7ea019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:38.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7ea6600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea010ab40 0x7f7ea01a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:38.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7ea6600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea0114b20 con 0x7f7ea0108f90 2024-04-26T20:09:38.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7ea6600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea01149a0 con 0x7f7ea010a180 2024-04-26T20:09:38.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7ea6600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea0114ca0 con 0x7f7ea010ab40 2024-04-26T20:09:38.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7e9f400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ea0108f90 0x7f7ea019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:38.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7e9fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea010ab40 0x7f7ea01a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:38.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7e97e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea010a180 0x7f7ea019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:38.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7e9f400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ea0108f90 0x7f7ea019f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55126/0 (socket says 172.21.15.94:55126) 2024-04-26T20:09:38.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7e9fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea010ab40 0x7f7ea01a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:37994/0 (socket says 172.21.15.94:37994) 2024-04-26T20:09:38.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.447+0000 7f7e9f400640 1 -- 172.21.15.94:0/2915897282 learned_addr learned my addr 172.21.15.94:0/2915897282 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:38.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e97e00640 1 -- 172.21.15.94:0/2915897282 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea010ab40 msgr2=0x7f7ea01a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e97e00640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea010ab40 0x7f7ea01a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e97e00640 1 -- 172.21.15.94:0/2915897282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ea0108f90 msgr2=0x7f7ea019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:38.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e97e00640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ea0108f90 0x7f7ea019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:38.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e97e00640 1 -- 172.21.15.94:0/2915897282 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea01a6dc0 con 0x7f7ea010a180 2024-04-26T20:09:38.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e9fe00640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea010ab40 0x7f7ea01a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:09:39.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e97e00640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea010a180 0x7f7ea019f6e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800e9f0 tx=0x7f7e8800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:39.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e9cc00640 1 -- 172.21.15.94:0/2915897282 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8800cd80 con 0x7f7ea010a180 2024-04-26T20:09:39.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e9cc00640 1 -- 172.21.15.94:0/2915897282 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e880035f0 con 0x7f7ea010a180 2024-04-26T20:09:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7e9cc00640 1 -- 172.21.15.94:0/2915897282 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e88010670 con 0x7f7ea010a180 2024-04-26T20:09:39.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.448+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea01a70b0 con 0x7f7ea010a180 2024-04-26T20:09:39.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.449+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea01a75b0 con 0x7f7ea010a180 2024-04-26T20:09:39.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.450+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea01102b0 con 0x7f7ea010a180 2024-04-26T20:09:39.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.452+0000 7f7e9cc00640 1 -- 172.21.15.94:0/2915897282 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7e880037a0 con 0x7f7ea010a180 2024-04-26T20:09:39.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.453+0000 7f7e9cc00640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7e840a3cc0 0x7f7e840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:39.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.453+0000 7f7e9cc00640 1 -- 172.21.15.94:0/2915897282 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f7e880d0d30 con 0x7f7ea010a180 2024-04-26T20:09:39.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.454+0000 7f7e9f400640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7e840a3cc0 0x7f7e840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:39.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.454+0000 7f7e9f400640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7e840a3cc0 0x7f7e840a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7e90000c00 tx=0x7f7e90005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.455+0000 7f7e9cc00640 1 -- 172.21.15.94:0/2915897282 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7e88054060 con 0x7f7ea010a180 2024-04-26T20:09:39.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:38 smithi204 ceph-mon[32618]: pgmap v367: 129 pgs: 129 active+clean; 623 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-04-26T20:09:39.205 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:38 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2915897282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:39.213 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:38 smithi103 ceph-mon[32635]: pgmap v367: 129 pgs: 129 active+clean; 623 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-04-26T20:09:39.326 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:38 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2915897282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.657+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ea002dca0 con 0x7f7ea010a180 2024-04-26T20:09:39.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.659+0000 7f7e9cc00640 1 -- 172.21.15.94:0/2915897282 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24635 (secure 0 0 0) 0x7f7ea002dca0 con 0x7f7ea010a180 2024-04-26T20:09:39.387 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:39.403 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:09:13.256835+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:09:39.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7e840a3cc0 msgr2=0x7f7e840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:39.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7e840a3cc0 0x7f7e840a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7e90000c00 tx=0x7f7e90005a20 comp rx=0 tx=0).stop 2024-04-26T20:09:39.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea010a180 msgr2=0x7f7ea019f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:39.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea010a180 0x7f7ea019f6e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800e9f0 tx=0x7f7e8800eeb0 comp rx=0 tx=0).stop 2024-04-26T20:09:39.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 shutdown_connections 2024-04-26T20:09:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7e840a3cc0 0x7f7e840a6180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:39.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ea010ab40 0x7f7ea01a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:39.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea010a180 0x7f7ea019f6e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:39.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 --2- 172.21.15.94:0/2915897282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ea0108f90 0x7f7ea019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:39.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 >> 172.21.15.94:0/2915897282 conn(0x7f7ea0077f40 msgr2=0x7f7ea010d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 shutdown_connections 2024-04-26T20:09:39.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:38.662+0000 7f7ea6600640 1 -- 172.21.15.94:0/2915897282 wait complete. 2024-04-26T20:09:39.730 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T20:09:39.745 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T20:09:39.756 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:38 smithi094 ceph-mon[28050]: pgmap v367: 129 pgs: 129 active+clean; 623 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-04-26T20:09:39.785 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:38 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2915897282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:40.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.012+0000 7fc7afe00640 1 -- 172.21.15.94:0/262875155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b010a940 msgr2=0x7fc7b01111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:40.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.012+0000 7fc7afe00640 1 --2- 172.21.15.94:0/262875155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b010a940 0x7fc7b01111d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009ca0 tx=0x7fc7a802a8b0 comp rx=0 tx=0).stop 2024-04-26T20:09:40.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.012+0000 7fc7afe00640 1 -- 172.21.15.94:0/262875155 shutdown_connections 2024-04-26T20:09:40.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.012+0000 7fc7afe00640 1 --2- 172.21.15.94:0/262875155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b010a940 0x7fc7b01111d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.012+0000 7fc7afe00640 1 --2- 172.21.15.94:0/262875155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b0109f80 0x7fc7b010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:40.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.012+0000 7fc7afe00640 1 --2- 172.21.15.94:0/262875155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7b0108d90 0x7fc7b01091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:40.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.012+0000 7fc7afe00640 1 -- 172.21.15.94:0/262875155 >> 172.21.15.94:0/262875155 conn(0x7fc7b0077d30 msgr2=0x7fc7b007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:40.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.013+0000 7fc7afe00640 1 -- 172.21.15.94:0/262875155 shutdown_connections 2024-04-26T20:09:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.013+0000 7fc7afe00640 1 -- 172.21.15.94:0/262875155 wait complete. 2024-04-26T20:09:40.113 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:39 smithi103 ceph-mon[32635]: pgmap v368: 129 pgs: 129 active+clean; 623 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-04-26T20:09:40.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.013+0000 7fc7afe00640 1 Processor -- start 2024-04-26T20:09:40.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7afe00640 1 -- start start 2024-04-26T20:09:40.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7afe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b0108d90 0x7fc7b019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:40.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7afe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7b0109f80 0x7fc7b019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:40.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7afe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b010a940 0x7fc7b01a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:40.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7afe00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b01148c0 con 0x7fc7b0108d90 2024-04-26T20:09:40.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7afe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0114740 con 0x7fc7b0109f80 2024-04-26T20:09:40.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7afe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0114a40 con 0x7fc7b010a940 2024-04-26T20:09:40.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7aea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b0108d90 0x7fc7b019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:40.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7aea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b0108d90 0x7fc7b019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55146/0 (socket says 172.21.15.94:55146) 2024-04-26T20:09:40.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7aea00640 1 -- 172.21.15.94:0/684490340 learned_addr learned my addr 172.21.15.94:0/684490340 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:40.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7af400640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b010a940 0x7fc7b01a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:40.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.014+0000 7fc7ae000640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7b0109f80 0x7fc7b019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:40.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7aea00640 1 -- 172.21.15.94:0/684490340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b010a940 msgr2=0x7fc7b01a6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:40.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7aea00640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b010a940 0x7fc7b01a6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:40.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7aea00640 1 -- 172.21.15.94:0/684490340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7b0109f80 msgr2=0x7fc7b019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7aea00640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7b0109f80 0x7fc7b019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:40.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7aea00640 1 -- 172.21.15.94:0/684490340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b01a6bc0 con 0x7fc7b0108d90 2024-04-26T20:09:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7aea00640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b0108d90 0x7fc7b019eff0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000e9c0 tx=0x7fc7a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:40.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7a7400640 1 -- 172.21.15.94:0/684490340 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a000cde0 con 0x7fc7b0108d90 2024-04-26T20:09:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7a7400640 1 -- 172.21.15.94:0/684490340 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7a00108e0 con 0x7fc7b0108d90 2024-04-26T20:09:40.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b01a6eb0 con 0x7fc7b0108d90 2024-04-26T20:09:40.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7a7400640 1 -- 172.21.15.94:0/684490340 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0018960 con 0x7fc7b0108d90 2024-04-26T20:09:40.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.015+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b01a7310 con 0x7fc7b0108d90 2024-04-26T20:09:40.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.016+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b0110100 con 0x7fc7b0108d90 2024-04-26T20:09:40.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.018+0000 7fc7a7400640 1 -- 172.21.15.94:0/684490340 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc7a001f020 con 0x7fc7b0108d90 2024-04-26T20:09:40.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.020+0000 7fc7a7400640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc7940a3c40 0x7fc7940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:40.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.020+0000 7fc7a7400640 1 -- 172.21.15.94:0/684490340 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fc7a00d0bc0 con 0x7fc7b0108d90 2024-04-26T20:09:40.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.020+0000 7fc7ae000640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc7940a3c40 0x7fc7940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.021+0000 7fc7ae000640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc7940a3c40 0x7fc7940a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc7b01a04b0 tx=0x7fc79c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:40.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.024+0000 7fc7a7400640 1 -- 172.21.15.94:0/684490340 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc7a0054060 con 0x7fc7b0108d90 2024-04-26T20:09:40.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:39 smithi204 ceph-mon[32618]: pgmap v368: 129 pgs: 129 active+clean; 623 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-04-26T20:09:40.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.223+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc7b0062d70 con 0x7fc7940a3c40 2024-04-26T20:09:40.902 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:39 smithi094 ceph-mon[28050]: pgmap v368: 129 pgs: 129 active+clean; 623 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-04-26T20:09:40.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.256+0000 7fc7a7400640 1 -- 172.21.15.94:0/684490340 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340354 (secure 0 0 0) 0x7fc7b0062d70 con 0x7fc7940a3c40 2024-04-26T20:09:40.926 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-26T20:09:40.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc7940a3c40 msgr2=0x7fc7940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:40.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc7940a3c40 0x7fc7940a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc7b01a04b0 tx=0x7fc79c005f90 comp rx=0 tx=0).stop 2024-04-26T20:09:41.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b0108d90 msgr2=0x7fc7b019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:41.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b0108d90 0x7fc7b019eff0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000e9c0 tx=0x7fc7a000ee80 comp rx=0 tx=0).stop 2024-04-26T20:09:41.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 shutdown_connections 2024-04-26T20:09:41.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc7940a3c40 0x7fc7940a6100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:41.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b010a940 0x7fc7b01a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:41.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc7b0109f80 0x7fc7b019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:41.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 --2- 172.21.15.94:0/684490340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7b0108d90 0x7fc7b019eff0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:41.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 >> 172.21.15.94:0/684490340 conn(0x7fc7b0077d30 msgr2=0x7fc7b010d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:41.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 shutdown_connections 2024-04-26T20:09:41.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:40.263+0000 7fc7afe00640 1 -- 172.21.15.94:0/684490340 wait complete. 2024-04-26T20:09:41.196 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-04-26T20:09:39.591094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637944,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":225,"num_read_kb":208,"num_write":339,"num_write_kb":3034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392796,"kb_used_data":14524,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549028,"statfs":{"total":1151940427776,"available":1151538204672,"internally_reserved":0,"allocated":14872576,"data_stored":8209044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":387233084},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40746,"num_objects":9,"num_object_clones":0,"num_object_copies":29,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":86,"num_read_kb":84,"num_write":43,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005765"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311966+0000","last_change":"2024-04-26T20:08:17.653922+0000","last_active":"2024-04-26T20:09:13.311966+0000","last_peered":"2024-04-26T20:09:13.311966+0000","last_clean":"2024-04-26T20:09:13.311966+0000","last_became_active":"2024-04-26T20:08:17.653486+0000","last_became_peered":"2024-04-26T20:08:17.653486+0000","last_unstale":"2024-04-26T20:09:13.311966+0000","last_undegraded":"2024-04-26T20:09:13.311966+0000","last_fullsized":"2024-04-26T20:09:13.311966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:05.034829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269659+0000","last_change":"2024-04-26T20:08:17.650126+0000","last_active":"2024-04-26T20:09:13.269659+0000","last_peered":"2024-04-26T20:09:13.269659+0000","last_clean":"2024-04-26T20:09:13.269659+0000","last_became_active":"2024-04-26T20:08:17.649727+0000","last_became_peered":"2024-04-26T20:08:17.649727+0000","last_unstale":"2024-04-26T20:09:13.269659+0000","last_undegraded":"2024-04-26T20:09:13.269659+0000","last_fullsized":"2024-04-26T20:09:13.269659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:11:28.255286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270403+0000","last_change":"2024-04-26T20:08:17.646243+0000","last_active":"2024-04-26T20:09:13.270403+0000","last_peered":"2024-04-26T20:09:13.270403+0000","last_clean":"2024-04-26T20:09:13.270403+0000","last_became_active":"2024-04-26T20:08:17.645677+0000","last_became_peered":"2024-04-26T20:08:17.645677+0000","last_unstale":"2024-04-26T20:09:13.270403+0000","last_undegraded":"2024-04-26T20:09:13.270403+0000","last_fullsized":"2024-04-26T20:09:13.270403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:31.698329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272081+0000","last_change":"2024-04-26T20:08:17.639845+0000","last_active":"2024-04-26T20:09:13.272081+0000","last_peered":"2024-04-26T20:09:13.272081+0000","last_clean":"2024-04-26T20:09:13.272081+0000","last_became_active":"2024-04-26T20:08:17.639465+0000","last_became_peered":"2024-04-26T20:08:17.639465+0000","last_unstale":"2024-04-26T20:09:13.272081+0000","last_undegraded":"2024-04-26T20:09:13.272081+0000","last_fullsized":"2024-04-26T20:09:13.272081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:05.717196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271298+0000","last_change":"2024-04-26T20:08:17.660459+0000","last_active":"2024-04-26T20:09:13.271298+0000","last_peered":"2024-04-26T20:09:13.271298+0000","last_clean":"2024-04-26T20:09:13.271298+0000","last_became_active":"2024-04-26T20:08:17.659927+0000","last_became_peered":"2024-04-26T20:08:17.659927+0000","last_unstale":"2024-04-26T20:09:13.271298+0000","last_undegraded":"2024-04-26T20:09:13.271298+0000","last_fullsized":"2024-04-26T20:09:13.271298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:10.062073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279825+0000","last_change":"2024-04-26T20:09:13.279825+0000","last_active":"2024-04-26T20:09:13.279825+0000","last_peered":"2024-04-26T20:09:13.279825+0000","last_clean":"2024-04-26T20:09:13.279825+0000","last_became_active":"2024-04-26T20:09:13.279464+0000","last_became_peered":"2024-04-26T20:09:13.279464+0000","last_unstale":"2024-04-26T20:09:13.279825+0000","last_undegraded":"2024-04-26T20:09:13.279825+0000","last_fullsized":"2024-04-26T20:09:13.279825+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:41.687987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312677+0000","last_change":"2024-04-26T20:08:17.648806+0000","last_active":"2024-04-26T20:09:13.312677+0000","last_peered":"2024-04-26T20:09:13.312677+0000","last_clean":"2024-04-26T20:09:13.312677+0000","last_became_active":"2024-04-26T20:08:17.648552+0000","last_became_peered":"2024-04-26T20:08:17.648552+0000","last_unstale":"2024-04-26T20:09:13.312677+0000","last_undegraded":"2024-04-26T20:09:13.312677+0000","last_fullsized":"2024-04-26T20:09:13.312677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:43.282610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313087+0000","last_change":"2024-04-26T20:08:17.663479+0000","last_active":"2024-04-26T20:09:13.313087+0000","last_peered":"2024-04-26T20:09:13.313087+0000","last_clean":"2024-04-26T20:09:13.313087+0000","last_became_active":"2024-04-26T20:08:17.663043+0000","last_became_peered":"2024-04-26T20:08:17.663043+0000","last_unstale":"2024-04-26T20:09:13.313087+0000","last_undegraded":"2024-04-26T20:09:13.313087+0000","last_fullsized":"2024-04-26T20:09:13.313087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:21:36.518513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313828+0000","last_change":"2024-04-26T20:08:17.637978+0000","last_active":"2024-04-26T20:09:13.313828+0000","last_peered":"2024-04-26T20:09:13.313828+0000","last_clean":"2024-04-26T20:09:13.313828+0000","last_became_active":"2024-04-26T20:08:17.637458+0000","last_became_peered":"2024-04-26T20:08:17.637458+0000","last_unstale":"2024-04-26T20:09:13.313828+0000","last_undegraded":"2024-04-26T20:09:13.313828+0000","last_fullsized":"2024-04-26T20:09:13.313828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:20:01.232741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.274941+0000","last_change":"2024-04-26T20:08:17.639819+0000","last_active":"2024-04-26T20:09:13.274941+0000","last_peered":"2024-04-26T20:09:13.274941+0000","last_clean":"2024-04-26T20:09:13.274941+0000","last_became_active":"2024-04-26T20:08:17.639479+0000","last_became_peered":"2024-04-26T20:08:17.639479+0000","last_unstale":"2024-04-26T20:09:13.274941+0000","last_undegraded":"2024-04-26T20:09:13.274941+0000","last_fullsized":"2024-04-26T20:09:13.274941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:44.536585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313378+0000","last_change":"2024-04-26T20:08:17.642533+0000","last_active":"2024-04-26T20:09:13.313378+0000","last_peered":"2024-04-26T20:09:13.313378+0000","last_clean":"2024-04-26T20:09:13.313378+0000","last_became_active":"2024-04-26T20:08:17.642198+0000","last_became_peered":"2024-04-26T20:08:17.642198+0000","last_unstale":"2024-04-26T20:09:13.313378+0000","last_undegraded":"2024-04-26T20:09:13.313378+0000","last_fullsized":"2024-04-26T20:09:13.313378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:16:21.200183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271602+0000","last_change":"2024-04-26T20:08:17.644646+0000","last_active":"2024-04-26T20:09:13.271602+0000","last_peered":"2024-04-26T20:09:13.271602+0000","last_clean":"2024-04-26T20:09:13.271602+0000","last_became_active":"2024-04-26T20:08:17.644314+0000","last_became_peered":"2024-04-26T20:08:17.644314+0000","last_unstale":"2024-04-26T20:09:13.271602+0000","last_undegraded":"2024-04-26T20:09:13.271602+0000","last_fullsized":"2024-04-26T20:09:13.271602+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:55:00.003120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279260+0000","last_change":"2024-04-26T20:08:17.657118+0000","last_active":"2024-04-26T20:09:13.279260+0000","last_peered":"2024-04-26T20:09:13.279260+0000","last_clean":"2024-04-26T20:09:13.279260+0000","last_became_active":"2024-04-26T20:08:17.656256+0000","last_became_peered":"2024-04-26T20:08:17.656256+0000","last_unstale":"2024-04-26T20:09:13.279260+0000","last_undegraded":"2024-04-26T20:09:13.279260+0000","last_fullsized":"2024-04-26T20:09:13.279260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:09.133475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269622+0000","last_change":"2024-04-26T20:08:17.658234+0000","last_active":"2024-04-26T20:09:13.269622+0000","last_peered":"2024-04-26T20:09:13.269622+0000","last_clean":"2024-04-26T20:09:13.269622+0000","last_became_active":"2024-04-26T20:08:17.651714+0000","last_became_peered":"2024-04-26T20:08:17.651714+0000","last_unstale":"2024-04-26T20:09:13.269622+0000","last_undegraded":"2024-04-26T20:09:13.269622+0000","last_fullsized":"2024-04-26T20:09:13.269622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:30.630320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269216+0000","last_change":"2024-04-26T20:08:17.647898+0000","last_active":"2024-04-26T20:09:13.269216+0000","last_peered":"2024-04-26T20:09:13.269216+0000","last_clean":"2024-04-26T20:09:13.269216+0000","last_became_active":"2024-04-26T20:08:17.647463+0000","last_became_peered":"2024-04-26T20:08:17.647463+0000","last_unstale":"2024-04-26T20:09:13.269216+0000","last_undegraded":"2024-04-26T20:09:13.269216+0000","last_fullsized":"2024-04-26T20:09:13.269216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:24.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311681+0000","last_change":"2024-04-26T20:08:17.639518+0000","last_active":"2024-04-26T20:09:13.311681+0000","last_peered":"2024-04-26T20:09:13.311681+0000","last_clean":"2024-04-26T20:09:13.311681+0000","last_became_active":"2024-04-26T20:08:17.639183+0000","last_became_peered":"2024-04-26T20:08:17.639183+0000","last_unstale":"2024-04-26T20:09:13.311681+0000","last_undegraded":"2024-04-26T20:09:13.311681+0000","last_fullsized":"2024-04-26T20:09:13.311681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:41:12.893323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311825+0000","last_change":"2024-04-26T20:08:17.653956+0000","last_active":"2024-04-26T20:09:13.311825+0000","last_peered":"2024-04-26T20:09:13.311825+0000","last_clean":"2024-04-26T20:09:13.311825+0000","last_became_active":"2024-04-26T20:08:17.653576+0000","last_became_peered":"2024-04-26T20:08:17.653576+0000","last_unstale":"2024-04-26T20:09:13.311825+0000","last_undegraded":"2024-04-26T20:09:13.311825+0000","last_fullsized":"2024-04-26T20:09:13.311825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:35:09.538194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.281786+0000","last_change":"2024-04-26T20:08:17.649500+0000","last_active":"2024-04-26T20:09:13.281786+0000","last_peered":"2024-04-26T20:09:13.281786+0000","last_clean":"2024-04-26T20:09:13.281786+0000","last_became_active":"2024-04-26T20:08:17.649062+0000","last_became_peered":"2024-04-26T20:08:17.649062+0000","last_unstale":"2024-04-26T20:09:13.281786+0000","last_undegraded":"2024-04-26T20:09:13.281786+0000","last_fullsized":"2024-04-26T20:09:13.281786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:24.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268715+0000","last_change":"2024-04-26T20:08:17.666440+0000","last_active":"2024-04-26T20:09:13.268715+0000","last_peered":"2024-04-26T20:09:13.268715+0000","last_clean":"2024-04-26T20:09:13.268715+0000","last_became_active":"2024-04-26T20:08:17.666024+0000","last_became_peered":"2024-04-26T20:08:17.666024+0000","last_unstale":"2024-04-26T20:09:13.268715+0000","last_undegraded":"2024-04-26T20:09:13.268715+0000","last_fullsized":"2024-04-26T20:09:13.268715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:55.356424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268092+0000","last_change":"2024-04-26T20:08:25.700728+0000","last_active":"2024-04-26T20:09:13.268092+0000","last_peered":"2024-04-26T20:09:13.268092+0000","last_clean":"2024-04-26T20:09:13.268092+0000","last_became_active":"2024-04-26T20:08:25.700339+0000","last_became_peered":"2024-04-26T20:08:25.700339+0000","last_unstale":"2024-04-26T20:09:13.268092+0000","last_undegraded":"2024-04-26T20:09:13.268092+0000","last_fullsized":"2024-04-26T20:09:13.268092+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:53.581083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.363260+0000","last_change":"2024-04-26T20:08:15.632803+0000","last_active":"2024-04-26T20:09:23.363260+0000","last_peered":"2024-04-26T20:09:23.363260+0000","last_clean":"2024-04-26T20:09:23.363260+0000","last_became_active":"2024-04-26T20:08:15.632368+0000","last_became_peered":"2024-04-26T20:08:15.632368+0000","last_unstale":"2024-04-26T20:09:23.363260+0000","last_undegraded":"2024-04-26T20:09:23.363260+0000","last_fullsized":"2024-04-26T20:09:23.363260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:55:54.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.315743+0000","last_change":"2024-04-26T20:08:17.657637+0000","last_active":"2024-04-26T20:09:13.315743+0000","last_peered":"2024-04-26T20:09:13.315743+0000","last_clean":"2024-04-26T20:09:13.315743+0000","last_became_active":"2024-04-26T20:08:17.657290+0000","last_became_peered":"2024-04-26T20:08:17.657290+0000","last_unstale":"2024-04-26T20:09:13.315743+0000","last_undegraded":"2024-04-26T20:09:13.315743+0000","last_fullsized":"2024-04-26T20:09:13.315743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:25.270989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.318084+0000","last_change":"2024-04-26T20:09:13.318084+0000","last_active":"2024-04-26T20:09:13.318084+0000","last_peered":"2024-04-26T20:09:13.318084+0000","last_clean":"2024-04-26T20:09:13.318084+0000","last_became_active":"2024-04-26T20:09:13.317689+0000","last_became_peered":"2024-04-26T20:09:13.317689+0000","last_unstale":"2024-04-26T20:09:13.318084+0000","last_undegraded":"2024-04-26T20:09:13.318084+0000","last_fullsized":"2024-04-26T20:09:13.318084+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:48.182186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.316341+0000","last_change":"2024-04-26T20:08:16.128717+0000","last_active":"2024-04-26T20:09:13.316341+0000","last_peered":"2024-04-26T20:09:13.316341+0000","last_clean":"2024-04-26T20:09:13.316341+0000","last_became_active":"2024-04-26T20:08:16.127738+0000","last_became_peered":"2024-04-26T20:08:16.127738+0000","last_unstale":"2024-04-26T20:09:13.316341+0000","last_undegraded":"2024-04-26T20:09:13.316341+0000","last_fullsized":"2024-04-26T20:09:13.316341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:01:26.343914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270890+0000","last_change":"2024-04-26T20:08:17.659613+0000","last_active":"2024-04-26T20:09:13.270890+0000","last_peered":"2024-04-26T20:09:13.270890+0000","last_clean":"2024-04-26T20:09:13.270890+0000","last_became_active":"2024-04-26T20:08:17.658750+0000","last_became_peered":"2024-04-26T20:08:17.658750+0000","last_unstale":"2024-04-26T20:09:13.270890+0000","last_undegraded":"2024-04-26T20:09:13.270890+0000","last_fullsized":"2024-04-26T20:09:13.270890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:59:08.724656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271319+0000","last_change":"2024-04-26T20:08:25.709115+0000","last_active":"2024-04-26T20:09:13.271319+0000","last_peered":"2024-04-26T20:09:13.271319+0000","last_clean":"2024-04-26T20:09:13.271319+0000","last_became_active":"2024-04-26T20:08:25.708552+0000","last_became_peered":"2024-04-26T20:08:25.708552+0000","last_unstale":"2024-04-26T20:09:13.271319+0000","last_undegraded":"2024-04-26T20:09:13.271319+0000","last_fullsized":"2024-04-26T20:09:13.271319+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:20.735727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"94'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.353071+0000","last_change":"2024-04-26T20:08:15.624441+0000","last_active":"2024-04-26T20:09:23.353071+0000","last_peered":"2024-04-26T20:09:23.353071+0000","last_clean":"2024-04-26T20:09:23.353071+0000","last_became_active":"2024-04-26T20:08:15.623999+0000","last_became_peered":"2024-04-26T20:08:15.623999+0000","last_unstale":"2024-04-26T20:09:23.353071+0000","last_undegraded":"2024-04-26T20:09:23.353071+0000","last_fullsized":"2024-04-26T20:09:23.353071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:34.276360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311954+0000","last_change":"2024-04-26T20:08:17.656088+0000","last_active":"2024-04-26T20:09:13.311954+0000","last_peered":"2024-04-26T20:09:13.311954+0000","last_clean":"2024-04-26T20:09:13.311954+0000","last_became_active":"2024-04-26T20:08:17.655736+0000","last_became_peered":"2024-04-26T20:08:17.655736+0000","last_unstale":"2024-04-26T20:09:13.311954+0000","last_undegraded":"2024-04-26T20:09:13.311954+0000","last_fullsized":"2024-04-26T20:09:13.311954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:29.187893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270667+0000","last_change":"2024-04-26T20:08:25.703015+0000","last_active":"2024-04-26T20:09:13.270667+0000","last_peered":"2024-04-26T20:09:13.270667+0000","last_clean":"2024-04-26T20:09:13.270667+0000","last_became_active":"2024-04-26T20:08:25.702699+0000","last_became_peered":"2024-04-26T20:08:25.702699+0000","last_unstale":"2024-04-26T20:09:13.270667+0000","last_undegraded":"2024-04-26T20:09:13.270667+0000","last_fullsized":"2024-04-26T20:09:13.270667+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:29:17.082803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271542+0000","last_change":"2024-04-26T20:08:15.624092+0000","last_active":"2024-04-26T20:09:13.271542+0000","last_peered":"2024-04-26T20:09:13.271542+0000","last_clean":"2024-04-26T20:09:13.271542+0000","last_became_active":"2024-04-26T20:08:15.623540+0000","last_became_peered":"2024-04-26T20:08:15.623540+0000","last_unstale":"2024-04-26T20:09:13.271542+0000","last_undegraded":"2024-04-26T20:09:13.271542+0000","last_fullsized":"2024-04-26T20:09:13.271542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:06:17.162854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313449+0000","last_change":"2024-04-26T20:09:13.313449+0000","last_active":"2024-04-26T20:09:13.313449+0000","last_peered":"2024-04-26T20:09:13.313449+0000","last_clean":"2024-04-26T20:09:13.313449+0000","last_became_active":"2024-04-26T20:09:13.313124+0000","last_became_peered":"2024-04-26T20:09:13.313124+0000","last_unstale":"2024-04-26T20:09:13.313449+0000","last_undegraded":"2024-04-26T20:09:13.313449+0000","last_fullsized":"2024-04-26T20:09:13.313449+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:10.447987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268370+0000","last_change":"2024-04-26T20:08:25.684973+0000","last_active":"2024-04-26T20:09:13.268370+0000","last_peered":"2024-04-26T20:09:13.268370+0000","last_clean":"2024-04-26T20:09:13.268370+0000","last_became_active":"2024-04-26T20:08:25.684636+0000","last_became_peered":"2024-04-26T20:08:25.684636+0000","last_unstale":"2024-04-26T20:09:13.268370+0000","last_undegraded":"2024-04-26T20:09:13.268370+0000","last_fullsized":"2024-04-26T20:09:13.268370+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:47:17.140765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:36.558738+0000","last_change":"2024-04-26T20:08:15.626792+0000","last_active":"2024-04-26T20:09:36.558738+0000","last_peered":"2024-04-26T20:09:36.558738+0000","last_clean":"2024-04-26T20:09:36.558738+0000","last_became_active":"2024-04-26T20:08:15.626453+0000","last_became_peered":"2024-04-26T20:08:15.626453+0000","last_unstale":"2024-04-26T20:09:36.558738+0000","last_undegraded":"2024-04-26T20:09:36.558738+0000","last_fullsized":"2024-04-26T20:09:36.558738+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:27.835127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271204+0000","last_change":"2024-04-26T20:08:17.660345+0000","last_active":"2024-04-26T20:09:13.271204+0000","last_peered":"2024-04-26T20:09:13.271204+0000","last_clean":"2024-04-26T20:09:13.271204+0000","last_became_active":"2024-04-26T20:08:17.659966+0000","last_became_peered":"2024-04-26T20:08:17.659966+0000","last_unstale":"2024-04-26T20:09:13.271204+0000","last_undegraded":"2024-04-26T20:09:13.271204+0000","last_fullsized":"2024-04-26T20:09:13.271204+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:48.762984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268348+0000","last_change":"2024-04-26T20:08:25.698002+0000","last_active":"2024-04-26T20:09:13.268348+0000","last_peered":"2024-04-26T20:09:13.268348+0000","last_clean":"2024-04-26T20:09:13.268348+0000","last_became_active":"2024-04-26T20:08:25.697588+0000","last_became_peered":"2024-04-26T20:08:25.697588+0000","last_unstale":"2024-04-26T20:09:13.268348+0000","last_undegraded":"2024-04-26T20:09:13.268348+0000","last_fullsized":"2024-04-26T20:09:13.268348+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:59.771941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.353673+0000","last_change":"2024-04-26T20:08:15.633759+0000","last_active":"2024-04-26T20:09:23.353673+0000","last_peered":"2024-04-26T20:09:23.353673+0000","last_clean":"2024-04-26T20:09:23.353673+0000","last_became_active":"2024-04-26T20:08:15.633354+0000","last_became_peered":"2024-04-26T20:08:15.633354+0000","last_unstale":"2024-04-26T20:09:23.353673+0000","last_undegraded":"2024-04-26T20:09:23.353673+0000","last_fullsized":"2024-04-26T20:09:23.353673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:33.394418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.319471+0000","last_change":"2024-04-26T20:09:13.319471+0000","last_active":"2024-04-26T20:09:13.319471+0000","last_peered":"2024-04-26T20:09:13.319471+0000","last_clean":"2024-04-26T20:09:13.319471+0000","last_became_active":"2024-04-26T20:09:13.318292+0000","last_became_peered":"2024-04-26T20:09:13.318292+0000","last_unstale":"2024-04-26T20:09:13.319471+0000","last_undegraded":"2024-04-26T20:09:13.319471+0000","last_fullsized":"2024-04-26T20:09:13.319471+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:41:36.278298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311786+0000","last_change":"2024-04-26T20:08:25.713338+0000","last_active":"2024-04-26T20:09:13.311786+0000","last_peered":"2024-04-26T20:09:13.311786+0000","last_clean":"2024-04-26T20:09:13.311786+0000","last_became_active":"2024-04-26T20:08:25.712905+0000","last_became_peered":"2024-04-26T20:08:25.712905+0000","last_unstale":"2024-04-26T20:09:13.311786+0000","last_undegraded":"2024-04-26T20:09:13.311786+0000","last_fullsized":"2024-04-26T20:09:13.311786+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:54.811957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.363336+0000","last_change":"2024-04-26T20:08:15.633813+0000","last_active":"2024-04-26T20:09:23.363336+0000","last_peered":"2024-04-26T20:09:23.363336+0000","last_clean":"2024-04-26T20:09:23.363336+0000","last_became_active":"2024-04-26T20:08:15.633448+0000","last_became_peered":"2024-04-26T20:08:15.633448+0000","last_unstale":"2024-04-26T20:09:23.363336+0000","last_undegraded":"2024-04-26T20:09:23.363336+0000","last_fullsized":"2024-04-26T20:09:23.363336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:52.909106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313267+0000","last_change":"2024-04-26T20:08:17.659606+0000","last_active":"2024-04-26T20:09:13.313267+0000","last_peered":"2024-04-26T20:09:13.313267+0000","last_clean":"2024-04-26T20:09:13.313267+0000","last_became_active":"2024-04-26T20:08:17.658948+0000","last_became_peered":"2024-04-26T20:08:17.658948+0000","last_unstale":"2024-04-26T20:09:13.313267+0000","last_undegraded":"2024-04-26T20:09:13.313267+0000","last_fullsized":"2024-04-26T20:09:13.313267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:03.616172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269511+0000","last_change":"2024-04-26T20:08:25.700149+0000","last_active":"2024-04-26T20:09:13.269511+0000","last_peered":"2024-04-26T20:09:13.269511+0000","last_clean":"2024-04-26T20:09:13.269511+0000","last_became_active":"2024-04-26T20:08:25.699693+0000","last_became_peered":"2024-04-26T20:08:25.699693+0000","last_unstale":"2024-04-26T20:09:13.269511+0000","last_undegraded":"2024-04-26T20:09:13.269511+0000","last_fullsized":"2024-04-26T20:09:13.269511+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:53.029443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"94'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271791+0000","last_change":"2024-04-26T20:08:15.626386+0000","last_active":"2024-04-26T20:09:13.271791+0000","last_peered":"2024-04-26T20:09:13.271791+0000","last_clean":"2024-04-26T20:09:13.271791+0000","last_became_active":"2024-04-26T20:08:15.626071+0000","last_became_peered":"2024-04-26T20:08:15.626071+0000","last_unstale":"2024-04-26T20:09:13.271791+0000","last_undegraded":"2024-04-26T20:09:13.271791+0000","last_fullsized":"2024-04-26T20:09:13.271791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:49.478038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.276617+0000","last_change":"2024-04-26T20:08:17.668616+0000","last_active":"2024-04-26T20:09:13.276617+0000","last_peered":"2024-04-26T20:09:13.276617+0000","last_clean":"2024-04-26T20:09:13.276617+0000","last_became_active":"2024-04-26T20:08:17.668210+0000","last_became_peered":"2024-04-26T20:08:17.668210+0000","last_unstale":"2024-04-26T20:09:13.276617+0000","last_undegraded":"2024-04-26T20:09:13.276617+0000","last_fullsized":"2024-04-26T20:09:13.276617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:42.219107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313045+0000","last_change":"2024-04-26T20:08:25.699132+0000","last_active":"2024-04-26T20:09:13.313045+0000","last_peered":"2024-04-26T20:09:13.313045+0000","last_clean":"2024-04-26T20:09:13.313045+0000","last_became_active":"2024-04-26T20:08:25.698675+0000","last_became_peered":"2024-04-26T20:08:25.698675+0000","last_unstale":"2024-04-26T20:09:13.313045+0000","last_undegraded":"2024-04-26T20:09:13.313045+0000","last_fullsized":"2024-04-26T20:09:13.313045+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:08:21.950765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'3","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:33.553744+0000","last_change":"2024-04-26T20:08:15.627167+0000","last_active":"2024-04-26T20:09:33.553744+0000","last_peered":"2024-04-26T20:09:33.553744+0000","last_clean":"2024-04-26T20:09:33.553744+0000","last_became_active":"2024-04-26T20:08:15.625928+0000","last_became_peered":"2024-04-26T20:08:15.625928+0000","last_unstale":"2024-04-26T20:09:33.553744+0000","last_undegraded":"2024-04-26T20:09:33.553744+0000","last_fullsized":"2024-04-26T20:09:33.553744+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:17:50.361864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313687+0000","last_change":"2024-04-26T20:08:17.652724+0000","last_active":"2024-04-26T20:09:13.313687+0000","last_peered":"2024-04-26T20:09:13.313687+0000","last_clean":"2024-04-26T20:09:13.313687+0000","last_became_active":"2024-04-26T20:08:17.652342+0000","last_became_peered":"2024-04-26T20:08:17.652342+0000","last_unstale":"2024-04-26T20:09:13.313687+0000","last_undegraded":"2024-04-26T20:09:13.313687+0000","last_fullsized":"2024-04-26T20:09:13.313687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:04:37.633017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.274813+0000","last_change":"2024-04-26T20:08:25.703811+0000","last_active":"2024-04-26T20:09:13.274813+0000","last_peered":"2024-04-26T20:09:13.274813+0000","last_clean":"2024-04-26T20:09:13.274813+0000","last_became_active":"2024-04-26T20:08:25.703247+0000","last_became_peered":"2024-04-26T20:08:25.703247+0000","last_unstale":"2024-04-26T20:09:13.274813+0000","last_undegraded":"2024-04-26T20:09:13.274813+0000","last_fullsized":"2024-04-26T20:09:13.274813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:19.605775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.345826+0000","last_change":"2024-04-26T20:08:15.628275+0000","last_active":"2024-04-26T20:09:22.345826+0000","last_peered":"2024-04-26T20:09:22.345826+0000","last_clean":"2024-04-26T20:09:22.345826+0000","last_became_active":"2024-04-26T20:08:15.627115+0000","last_became_peered":"2024-04-26T20:08:15.627115+0000","last_unstale":"2024-04-26T20:09:22.345826+0000","last_undegraded":"2024-04-26T20:09:22.345826+0000","last_fullsized":"2024-04-26T20:09:22.345826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:48.799646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270896+0000","last_change":"2024-04-26T20:08:17.651666+0000","last_active":"2024-04-26T20:09:13.270896+0000","last_peered":"2024-04-26T20:09:13.270896+0000","last_clean":"2024-04-26T20:09:13.270896+0000","last_became_active":"2024-04-26T20:08:17.644762+0000","last_became_peered":"2024-04-26T20:08:17.644762+0000","last_unstale":"2024-04-26T20:09:13.270896+0000","last_undegraded":"2024-04-26T20:09:13.270896+0000","last_fullsized":"2024-04-26T20:09:13.270896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:19.070044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.314326+0000","last_change":"2024-04-26T20:09:13.314326+0000","last_active":"2024-04-26T20:09:13.314326+0000","last_peered":"2024-04-26T20:09:13.314326+0000","last_clean":"2024-04-26T20:09:13.314326+0000","last_became_active":"2024-04-26T20:09:13.313823+0000","last_became_peered":"2024-04-26T20:09:13.313823+0000","last_unstale":"2024-04-26T20:09:13.314326+0000","last_undegraded":"2024-04-26T20:09:13.314326+0000","last_fullsized":"2024-04-26T20:09:13.314326+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:12:55.398462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.346533+0000","last_change":"2024-04-26T20:08:16.123015+0000","last_active":"2024-04-26T20:09:22.346533+0000","last_peered":"2024-04-26T20:09:22.346533+0000","last_clean":"2024-04-26T20:09:22.346533+0000","last_became_active":"2024-04-26T20:08:16.122594+0000","last_became_peered":"2024-04-26T20:08:16.122594+0000","last_unstale":"2024-04-26T20:09:22.346533+0000","last_undegraded":"2024-04-26T20:09:22.346533+0000","last_fullsized":"2024-04-26T20:09:22.346533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:54:11.846029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312071+0000","last_change":"2024-04-26T20:08:17.640882+0000","last_active":"2024-04-26T20:09:13.312071+0000","last_peered":"2024-04-26T20:09:13.312071+0000","last_clean":"2024-04-26T20:09:13.312071+0000","last_became_active":"2024-04-26T20:08:17.640399+0000","last_became_peered":"2024-04-26T20:08:17.640399+0000","last_unstale":"2024-04-26T20:09:13.312071+0000","last_undegraded":"2024-04-26T20:09:13.312071+0000","last_fullsized":"2024-04-26T20:09:13.312071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:05:57.554751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.275493+0000","last_change":"2024-04-26T20:08:25.713003+0000","last_active":"2024-04-26T20:09:13.275493+0000","last_peered":"2024-04-26T20:09:13.275493+0000","last_clean":"2024-04-26T20:09:13.275493+0000","last_became_active":"2024-04-26T20:08:25.711116+0000","last_became_peered":"2024-04-26T20:08:25.711116+0000","last_unstale":"2024-04-26T20:09:13.275493+0000","last_undegraded":"2024-04-26T20:09:13.275493+0000","last_fullsized":"2024-04-26T20:09:13.275493+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:45:50.627388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'26","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:34.556046+0000","last_change":"2024-04-26T20:08:15.629344+0000","last_active":"2024-04-26T20:09:34.556046+0000","last_peered":"2024-04-26T20:09:34.556046+0000","last_clean":"2024-04-26T20:09:34.556046+0000","last_became_active":"2024-04-26T20:08:15.628836+0000","last_became_peered":"2024-04-26T20:08:15.628836+0000","last_unstale":"2024-04-26T20:09:34.556046+0000","last_undegraded":"2024-04-26T20:09:34.556046+0000","last_fullsized":"2024-04-26T20:09:34.556046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:05:27.011109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43341,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":56,"num_write":27,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271652+0000","last_change":"2024-04-26T20:08:17.641469+0000","last_active":"2024-04-26T20:09:13.271652+0000","last_peered":"2024-04-26T20:09:13.271652+0000","last_clean":"2024-04-26T20:09:13.271652+0000","last_became_active":"2024-04-26T20:08:17.641147+0000","last_became_peered":"2024-04-26T20:08:17.641147+0000","last_unstale":"2024-04-26T20:09:13.271652+0000","last_undegraded":"2024-04-26T20:09:13.271652+0000","last_fullsized":"2024-04-26T20:09:13.271652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:36:02.885095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272784+0000","last_change":"2024-04-26T20:08:25.708459+0000","last_active":"2024-04-26T20:09:13.272784+0000","last_peered":"2024-04-26T20:09:13.272784+0000","last_clean":"2024-04-26T20:09:13.272784+0000","last_became_active":"2024-04-26T20:08:25.708099+0000","last_became_peered":"2024-04-26T20:08:25.708099+0000","last_unstale":"2024-04-26T20:09:13.272784+0000","last_undegraded":"2024-04-26T20:09:13.272784+0000","last_fullsized":"2024-04-26T20:09:13.272784+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:37.423731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"94'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.359007+0000","last_change":"2024-04-26T20:08:15.627399+0000","last_active":"2024-04-26T20:09:23.359007+0000","last_peered":"2024-04-26T20:09:23.359007+0000","last_clean":"2024-04-26T20:09:23.359007+0000","last_became_active":"2024-04-26T20:08:15.627018+0000","last_became_peered":"2024-04-26T20:08:15.627018+0000","last_unstale":"2024-04-26T20:09:23.359007+0000","last_undegraded":"2024-04-26T20:09:23.359007+0000","last_fullsized":"2024-04-26T20:09:23.359007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:13.491035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279474+0000","last_change":"2024-04-26T20:08:17.669883+0000","last_active":"2024-04-26T20:09:13.279474+0000","last_peered":"2024-04-26T20:09:13.279474+0000","last_clean":"2024-04-26T20:09:13.279474+0000","last_became_active":"2024-04-26T20:08:17.669484+0000","last_became_peered":"2024-04-26T20:08:17.669484+0000","last_unstale":"2024-04-26T20:09:13.279474+0000","last_undegraded":"2024-04-26T20:09:13.279474+0000","last_fullsized":"2024-04-26T20:09:13.279474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:53:28.820771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311593+0000","last_change":"2024-04-26T20:08:25.694352+0000","last_active":"2024-04-26T20:09:13.311593+0000","last_peered":"2024-04-26T20:09:13.311593+0000","last_clean":"2024-04-26T20:09:13.311593+0000","last_became_active":"2024-04-26T20:08:25.694022+0000","last_became_peered":"2024-04-26T20:08:25.694022+0000","last_unstale":"2024-04-26T20:09:13.311593+0000","last_undegraded":"2024-04-26T20:09:13.311593+0000","last_fullsized":"2024-04-26T20:09:13.311593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:23.830964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.345151+0000","last_change":"2024-04-26T20:08:15.625816+0000","last_active":"2024-04-26T20:09:22.345151+0000","last_peered":"2024-04-26T20:09:22.345151+0000","last_clean":"2024-04-26T20:09:22.345151+0000","last_became_active":"2024-04-26T20:08:15.625390+0000","last_became_peered":"2024-04-26T20:08:15.625390+0000","last_unstale":"2024-04-26T20:09:22.345151+0000","last_undegraded":"2024-04-26T20:09:22.345151+0000","last_fullsized":"2024-04-26T20:09:22.345151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:08:57.864795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312947+0000","last_change":"2024-04-26T20:08:17.653016+0000","last_active":"2024-04-26T20:09:13.312947+0000","last_peered":"2024-04-26T20:09:13.312947+0000","last_clean":"2024-04-26T20:09:13.312947+0000","last_became_active":"2024-04-26T20:08:17.652403+0000","last_became_peered":"2024-04-26T20:08:17.652403+0000","last_unstale":"2024-04-26T20:09:13.312947+0000","last_undegraded":"2024-04-26T20:09:13.312947+0000","last_fullsized":"2024-04-26T20:09:13.312947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:02.580052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.280597+0000","last_change":"2024-04-26T20:08:25.717469+0000","last_active":"2024-04-26T20:09:13.280597+0000","last_peered":"2024-04-26T20:09:13.280597+0000","last_clean":"2024-04-26T20:09:13.280597+0000","last_became_active":"2024-04-26T20:08:25.716701+0000","last_became_peered":"2024-04-26T20:08:25.716701+0000","last_unstale":"2024-04-26T20:09:13.280597+0000","last_undegraded":"2024-04-26T20:09:13.280597+0000","last_fullsized":"2024-04-26T20:09:13.280597+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:11:10.991196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'3","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:36.570721+0000","last_change":"2024-04-26T20:08:15.626268+0000","last_active":"2024-04-26T20:09:36.570721+0000","last_peered":"2024-04-26T20:09:36.570721+0000","last_clean":"2024-04-26T20:09:36.570721+0000","last_became_active":"2024-04-26T20:08:15.625839+0000","last_became_peered":"2024-04-26T20:08:15.625839+0000","last_unstale":"2024-04-26T20:09:36.570721+0000","last_undegraded":"2024-04-26T20:09:36.570721+0000","last_fullsized":"2024-04-26T20:09:36.570721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:54.537335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312557+0000","last_change":"2024-04-26T20:08:17.644651+0000","last_active":"2024-04-26T20:09:13.312557+0000","last_peered":"2024-04-26T20:09:13.312557+0000","last_clean":"2024-04-26T20:09:13.312557+0000","last_became_active":"2024-04-26T20:08:17.644357+0000","last_became_peered":"2024-04-26T20:08:17.644357+0000","last_unstale":"2024-04-26T20:09:13.312557+0000","last_undegraded":"2024-04-26T20:09:13.312557+0000","last_fullsized":"2024-04-26T20:09:13.312557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:36.344501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.278589+0000","last_change":"2024-04-26T20:08:25.710016+0000","last_active":"2024-04-26T20:09:13.278589+0000","last_peered":"2024-04-26T20:09:13.278589+0000","last_clean":"2024-04-26T20:09:13.278589+0000","last_became_active":"2024-04-26T20:08:25.709647+0000","last_became_peered":"2024-04-26T20:08:25.709647+0000","last_unstale":"2024-04-26T20:09:13.278589+0000","last_undegraded":"2024-04-26T20:09:13.278589+0000","last_fullsized":"2024-04-26T20:09:13.278589+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:22.434116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"94'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.361458+0000","last_change":"2024-04-26T20:08:16.128347+0000","last_active":"2024-04-26T20:09:23.361458+0000","last_peered":"2024-04-26T20:09:23.361458+0000","last_clean":"2024-04-26T20:09:23.361458+0000","last_became_active":"2024-04-26T20:08:16.127795+0000","last_became_peered":"2024-04-26T20:08:16.127795+0000","last_unstale":"2024-04-26T20:09:23.361458+0000","last_undegraded":"2024-04-26T20:09:23.361458+0000","last_fullsized":"2024-04-26T20:09:23.361458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:21.217202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.314298+0000","last_change":"2024-04-26T20:09:13.314298+0000","last_active":"2024-04-26T20:09:13.314298+0000","last_peered":"2024-04-26T20:09:13.314298+0000","last_clean":"2024-04-26T20:09:13.314298+0000","last_became_active":"2024-04-26T20:09:13.313732+0000","last_became_peered":"2024-04-26T20:09:13.313732+0000","last_unstale":"2024-04-26T20:09:13.314298+0000","last_undegraded":"2024-04-26T20:09:13.314298+0000","last_fullsized":"2024-04-26T20:09:13.314298+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:59:41.782682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272262+0000","last_change":"2024-04-26T20:08:25.700504+0000","last_active":"2024-04-26T20:09:13.272262+0000","last_peered":"2024-04-26T20:09:13.272262+0000","last_clean":"2024-04-26T20:09:13.272262+0000","last_became_active":"2024-04-26T20:08:25.700171+0000","last_became_peered":"2024-04-26T20:08:25.700171+0000","last_unstale":"2024-04-26T20:09:13.272262+0000","last_undegraded":"2024-04-26T20:09:13.272262+0000","last_fullsized":"2024-04-26T20:09:13.272262+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:04.442343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'2","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:24.350667+0000","last_change":"2024-04-26T20:08:15.625738+0000","last_active":"2024-04-26T20:09:24.350667+0000","last_peered":"2024-04-26T20:09:24.350667+0000","last_clean":"2024-04-26T20:09:24.350667+0000","last_became_active":"2024-04-26T20:08:15.625381+0000","last_became_peered":"2024-04-26T20:08:15.625381+0000","last_unstale":"2024-04-26T20:09:24.350667+0000","last_undegraded":"2024-04-26T20:09:24.350667+0000","last_fullsized":"2024-04-26T20:09:24.350667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:52.378477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311750+0000","last_change":"2024-04-26T20:08:17.646682+0000","last_active":"2024-04-26T20:09:13.311750+0000","last_peered":"2024-04-26T20:09:13.311750+0000","last_clean":"2024-04-26T20:09:13.311750+0000","last_became_active":"2024-04-26T20:08:17.646362+0000","last_became_peered":"2024-04-26T20:08:17.646362+0000","last_unstale":"2024-04-26T20:09:13.311750+0000","last_undegraded":"2024-04-26T20:09:13.311750+0000","last_fullsized":"2024-04-26T20:09:13.311750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:52.955883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271299+0000","last_change":"2024-04-26T20:08:25.704580+0000","last_active":"2024-04-26T20:09:13.271299+0000","last_peered":"2024-04-26T20:09:13.271299+0000","last_clean":"2024-04-26T20:09:13.271299+0000","last_became_active":"2024-04-26T20:08:25.704083+0000","last_became_peered":"2024-04-26T20:08:25.704083+0000","last_unstale":"2024-04-26T20:09:13.271299+0000","last_undegraded":"2024-04-26T20:09:13.271299+0000","last_fullsized":"2024-04-26T20:09:13.271299+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:59:33.409501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.275553+0000","last_change":"2024-04-26T20:08:16.131048+0000","last_active":"2024-04-26T20:09:13.275553+0000","last_peered":"2024-04-26T20:09:13.275553+0000","last_clean":"2024-04-26T20:09:13.275553+0000","last_became_active":"2024-04-26T20:08:16.129700+0000","last_became_peered":"2024-04-26T20:08:16.129700+0000","last_unstale":"2024-04-26T20:09:13.275553+0000","last_undegraded":"2024-04-26T20:09:13.275553+0000","last_fullsized":"2024-04-26T20:09:13.275553+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:59:34.814697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.274604+0000","last_change":"2024-04-26T20:08:17.669408+0000","last_active":"2024-04-26T20:09:13.274604+0000","last_peered":"2024-04-26T20:09:13.274604+0000","last_clean":"2024-04-26T20:09:13.274604+0000","last_became_active":"2024-04-26T20:08:17.669102+0000","last_became_peered":"2024-04-26T20:08:17.669102+0000","last_unstale":"2024-04-26T20:09:13.274604+0000","last_undegraded":"2024-04-26T20:09:13.274604+0000","last_fullsized":"2024-04-26T20:09:13.274604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:55:29.812387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279902+0000","last_change":"2024-04-26T20:08:25.692910+0000","last_active":"2024-04-26T20:09:13.279902+0000","last_peered":"2024-04-26T20:09:13.279902+0000","last_clean":"2024-04-26T20:09:13.279902+0000","last_became_active":"2024-04-26T20:08:25.692292+0000","last_became_peered":"2024-04-26T20:08:25.692292+0000","last_unstale":"2024-04-26T20:09:13.279902+0000","last_undegraded":"2024-04-26T20:09:13.279902+0000","last_fullsized":"2024-04-26T20:09:13.279902+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:42:19.483756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313499+0000","last_change":"2024-04-26T20:08:15.633948+0000","last_active":"2024-04-26T20:09:13.313499+0000","last_peered":"2024-04-26T20:09:13.313499+0000","last_clean":"2024-04-26T20:09:13.313499+0000","last_became_active":"2024-04-26T20:08:15.633531+0000","last_became_peered":"2024-04-26T20:08:15.633531+0000","last_unstale":"2024-04-26T20:09:13.313499+0000","last_undegraded":"2024-04-26T20:09:13.313499+0000","last_fullsized":"2024-04-26T20:09:13.313499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:13:15.587856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.274063+0000","last_change":"2024-04-26T20:08:17.663674+0000","last_active":"2024-04-26T20:09:13.274063+0000","last_peered":"2024-04-26T20:09:13.274063+0000","last_clean":"2024-04-26T20:09:13.274063+0000","last_became_active":"2024-04-26T20:08:17.663309+0000","last_became_peered":"2024-04-26T20:08:17.663309+0000","last_unstale":"2024-04-26T20:09:13.274063+0000","last_undegraded":"2024-04-26T20:09:13.274063+0000","last_fullsized":"2024-04-26T20:09:13.274063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:20.661026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279387+0000","last_change":"2024-04-26T20:08:25.698807+0000","last_active":"2024-04-26T20:09:13.279387+0000","last_peered":"2024-04-26T20:09:13.279387+0000","last_clean":"2024-04-26T20:09:13.279387+0000","last_became_active":"2024-04-26T20:08:25.697686+0000","last_became_peered":"2024-04-26T20:08:25.697686+0000","last_unstale":"2024-04-26T20:09:13.279387+0000","last_undegraded":"2024-04-26T20:09:13.279387+0000","last_fullsized":"2024-04-26T20:09:13.279387+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:59:37.099531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272356+0000","last_change":"2024-04-26T20:08:17.645055+0000","last_active":"2024-04-26T20:09:13.272356+0000","last_peered":"2024-04-26T20:09:13.272356+0000","last_clean":"2024-04-26T20:09:13.272356+0000","last_became_active":"2024-04-26T20:08:17.644715+0000","last_became_peered":"2024-04-26T20:08:17.644715+0000","last_unstale":"2024-04-26T20:09:13.272356+0000","last_undegraded":"2024-04-26T20:09:13.272356+0000","last_fullsized":"2024-04-26T20:09:13.272356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:12.508925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'4","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:35.567377+0000","last_change":"2024-04-26T20:08:16.123710+0000","last_active":"2024-04-26T20:09:35.567377+0000","last_peered":"2024-04-26T20:09:35.567377+0000","last_clean":"2024-04-26T20:09:35.567377+0000","last_became_active":"2024-04-26T20:08:16.123046+0000","last_became_peered":"2024-04-26T20:08:16.123046+0000","last_unstale":"2024-04-26T20:09:35.567377+0000","last_undegraded":"2024-04-26T20:09:35.567377+0000","last_fullsized":"2024-04-26T20:09:35.567377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:42.389448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"94'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313027+0000","last_change":"2024-04-26T20:08:25.698716+0000","last_active":"2024-04-26T20:09:13.313027+0000","last_peered":"2024-04-26T20:09:13.313027+0000","last_clean":"2024-04-26T20:09:13.313027+0000","last_became_active":"2024-04-26T20:08:25.698418+0000","last_became_peered":"2024-04-26T20:08:25.698418+0000","last_unstale":"2024-04-26T20:09:13.313027+0000","last_undegraded":"2024-04-26T20:09:13.313027+0000","last_fullsized":"2024-04-26T20:09:13.313027+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:56.290241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.315352+0000","last_change":"2024-04-26T20:08:17.656393+0000","last_active":"2024-04-26T20:09:13.315352+0000","last_peered":"2024-04-26T20:09:13.315352+0000","last_clean":"2024-04-26T20:09:13.315352+0000","last_became_active":"2024-04-26T20:08:17.655914+0000","last_became_peered":"2024-04-26T20:08:17.655914+0000","last_unstale":"2024-04-26T20:09:13.315352+0000","last_undegraded":"2024-04-26T20:09:13.315352+0000","last_fullsized":"2024-04-26T20:09:13.315352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:38:41.459519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"94'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:24.539090+0000","last_change":"2024-04-26T20:08:15.631983+0000","last_active":"2024-04-26T20:09:24.539090+0000","last_peered":"2024-04-26T20:09:24.539090+0000","last_clean":"2024-04-26T20:09:24.539090+0000","last_became_active":"2024-04-26T20:08:15.631625+0000","last_became_peered":"2024-04-26T20:08:15.631625+0000","last_unstale":"2024-04-26T20:09:24.539090+0000","last_undegraded":"2024-04-26T20:09:24.539090+0000","last_fullsized":"2024-04-26T20:09:24.539090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:49.264149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":358,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312356+0000","last_change":"2024-04-26T20:06:47.857115+0000","last_active":"2024-04-26T20:09:13.312356+0000","last_peered":"2024-04-26T20:09:13.312356+0000","last_clean":"2024-04-26T20:09:13.312356+0000","last_became_active":"2024-04-26T20:06:47.856652+0000","last_became_peered":"2024-04-26T20:06:47.856652+0000","last_unstale":"2024-04-26T20:09:13.312356+0000","last_undegraded":"2024-04-26T20:09:13.312356+0000","last_fullsized":"2024-04-26T20:09:13.312356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:02:53.903427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:02:53.903427+0000","last_clean_scrub_stamp":"2024-04-26T20:02:53.903427+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:35:14.359328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.283142+0000","last_change":"2024-04-26T20:09:13.283142+0000","last_active":"2024-04-26T20:09:13.283142+0000","last_peered":"2024-04-26T20:09:13.283142+0000","last_clean":"2024-04-26T20:09:13.283142+0000","last_became_active":"2024-04-26T20:09:13.282806+0000","last_became_peered":"2024-04-26T20:09:13.282806+0000","last_unstale":"2024-04-26T20:09:13.283142+0000","last_undegraded":"2024-04-26T20:09:13.283142+0000","last_fullsized":"2024-04-26T20:09:13.283142+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:39.595299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.278860+0000","last_change":"2024-04-26T20:08:17.654402+0000","last_active":"2024-04-26T20:09:13.278860+0000","last_peered":"2024-04-26T20:09:13.278860+0000","last_clean":"2024-04-26T20:09:13.278860+0000","last_became_active":"2024-04-26T20:08:17.650047+0000","last_became_peered":"2024-04-26T20:08:17.650047+0000","last_unstale":"2024-04-26T20:09:13.278860+0000","last_undegraded":"2024-04-26T20:09:13.278860+0000","last_fullsized":"2024-04-26T20:09:13.278860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:40.225984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.355000+0000","last_change":"2024-04-26T20:08:15.626610+0000","last_active":"2024-04-26T20:09:23.355000+0000","last_peered":"2024-04-26T20:09:23.355000+0000","last_clean":"2024-04-26T20:09:23.355000+0000","last_became_active":"2024-04-26T20:08:15.626162+0000","last_became_peered":"2024-04-26T20:08:15.626162+0000","last_unstale":"2024-04-26T20:09:23.355000+0000","last_undegraded":"2024-04-26T20:09:23.355000+0000","last_fullsized":"2024-04-26T20:09:23.355000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:20.270714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311569+0000","last_change":"2024-04-26T20:08:25.697123+0000","last_active":"2024-04-26T20:09:13.311569+0000","last_peered":"2024-04-26T20:09:13.311569+0000","last_clean":"2024-04-26T20:09:13.311569+0000","last_became_active":"2024-04-26T20:08:25.696773+0000","last_became_peered":"2024-04-26T20:08:25.696773+0000","last_unstale":"2024-04-26T20:09:13.311569+0000","last_undegraded":"2024-04-26T20:09:13.311569+0000","last_fullsized":"2024-04-26T20:09:13.311569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:22:56.642427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"94'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.345678+0000","last_change":"2024-04-26T20:08:16.128206+0000","last_active":"2024-04-26T20:09:22.345678+0000","last_peered":"2024-04-26T20:09:22.345678+0000","last_clean":"2024-04-26T20:09:22.345678+0000","last_became_active":"2024-04-26T20:08:16.127703+0000","last_became_peered":"2024-04-26T20:08:16.127703+0000","last_unstale":"2024-04-26T20:09:22.345678+0000","last_undegraded":"2024-04-26T20:09:22.345678+0000","last_fullsized":"2024-04-26T20:09:22.345678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:39.852292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.280079+0000","last_change":"2024-04-26T20:08:17.652810+0000","last_active":"2024-04-26T20:09:13.280079+0000","last_peered":"2024-04-26T20:09:13.280079+0000","last_clean":"2024-04-26T20:09:13.280079+0000","last_became_active":"2024-04-26T20:08:17.652486+0000","last_became_peered":"2024-04-26T20:08:17.652486+0000","last_unstale":"2024-04-26T20:09:13.280079+0000","last_undegraded":"2024-04-26T20:09:13.280079+0000","last_fullsized":"2024-04-26T20:09:13.280079+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:43.382814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313266+0000","last_change":"2024-04-26T20:08:25.701678+0000","last_active":"2024-04-26T20:09:13.313266+0000","last_peered":"2024-04-26T20:09:13.313266+0000","last_clean":"2024-04-26T20:09:13.313266+0000","last_became_active":"2024-04-26T20:08:25.701283+0000","last_became_peered":"2024-04-26T20:08:25.701283+0000","last_unstale":"2024-04-26T20:09:13.313266+0000","last_undegraded":"2024-04-26T20:09:13.313266+0000","last_fullsized":"2024-04-26T20:09:13.313266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:35:26.774248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:36.675403+0000","last_change":"2024-04-26T20:08:15.634215+0000","last_active":"2024-04-26T20:09:36.675403+0000","last_peered":"2024-04-26T20:09:36.675403+0000","last_clean":"2024-04-26T20:09:36.675403+0000","last_became_active":"2024-04-26T20:08:15.633669+0000","last_became_peered":"2024-04-26T20:08:15.633669+0000","last_unstale":"2024-04-26T20:09:36.675403+0000","last_undegraded":"2024-04-26T20:09:36.675403+0000","last_fullsized":"2024-04-26T20:09:36.675403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:06.178974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.316357+0000","last_change":"2024-04-26T20:08:17.666557+0000","last_active":"2024-04-26T20:09:13.316357+0000","last_peered":"2024-04-26T20:09:13.316357+0000","last_clean":"2024-04-26T20:09:13.316357+0000","last_became_active":"2024-04-26T20:08:17.665552+0000","last_became_peered":"2024-04-26T20:08:17.665552+0000","last_unstale":"2024-04-26T20:09:13.316357+0000","last_undegraded":"2024-04-26T20:09:13.316357+0000","last_fullsized":"2024-04-26T20:09:13.316357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:11.993087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268009+0000","last_change":"2024-04-26T20:08:25.684541+0000","last_active":"2024-04-26T20:09:13.268009+0000","last_peered":"2024-04-26T20:09:13.268009+0000","last_clean":"2024-04-26T20:09:13.268009+0000","last_became_active":"2024-04-26T20:08:25.684159+0000","last_became_peered":"2024-04-26T20:08:25.684159+0000","last_unstale":"2024-04-26T20:09:13.268009+0000","last_undegraded":"2024-04-26T20:09:13.268009+0000","last_fullsized":"2024-04-26T20:09:13.268009+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:31:45.405767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"94'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.343844+0000","last_change":"2024-04-26T20:08:16.002624+0000","last_active":"2024-04-26T20:09:22.343844+0000","last_peered":"2024-04-26T20:09:22.343844+0000","last_clean":"2024-04-26T20:09:22.343844+0000","last_became_active":"2024-04-26T20:08:16.002246+0000","last_became_peered":"2024-04-26T20:08:16.002246+0000","last_unstale":"2024-04-26T20:09:22.343844+0000","last_undegraded":"2024-04-26T20:09:22.343844+0000","last_fullsized":"2024-04-26T20:09:22.343844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:59.393586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.314961+0000","last_change":"2024-04-26T20:09:13.314961+0000","last_active":"2024-04-26T20:09:13.314961+0000","last_peered":"2024-04-26T20:09:13.314961+0000","last_clean":"2024-04-26T20:09:13.314961+0000","last_became_active":"2024-04-26T20:09:13.314533+0000","last_became_peered":"2024-04-26T20:09:13.314533+0000","last_unstale":"2024-04-26T20:09:13.314961+0000","last_undegraded":"2024-04-26T20:09:13.314961+0000","last_fullsized":"2024-04-26T20:09:13.314961+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:29:27.518198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271118+0000","last_change":"2024-04-26T20:08:25.713891+0000","last_active":"2024-04-26T20:09:13.271118+0000","last_peered":"2024-04-26T20:09:13.271118+0000","last_clean":"2024-04-26T20:09:13.271118+0000","last_became_active":"2024-04-26T20:08:25.713480+0000","last_became_peered":"2024-04-26T20:08:25.713480+0000","last_unstale":"2024-04-26T20:09:13.271118+0000","last_undegraded":"2024-04-26T20:09:13.271118+0000","last_fullsized":"2024-04-26T20:09:13.271118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:26.226811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.346453+0000","last_change":"2024-04-26T20:08:16.123332+0000","last_active":"2024-04-26T20:09:22.346453+0000","last_peered":"2024-04-26T20:09:22.346453+0000","last_clean":"2024-04-26T20:09:22.346453+0000","last_became_active":"2024-04-26T20:08:16.122944+0000","last_became_peered":"2024-04-26T20:08:16.122944+0000","last_unstale":"2024-04-26T20:09:22.346453+0000","last_undegraded":"2024-04-26T20:09:22.346453+0000","last_fullsized":"2024-04-26T20:09:22.346453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:37:23.369867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271089+0000","last_change":"2024-04-26T20:08:17.641693+0000","last_active":"2024-04-26T20:09:13.271089+0000","last_peered":"2024-04-26T20:09:13.271089+0000","last_clean":"2024-04-26T20:09:13.271089+0000","last_became_active":"2024-04-26T20:08:17.641334+0000","last_became_peered":"2024-04-26T20:08:17.641334+0000","last_unstale":"2024-04-26T20:09:13.271089+0000","last_undegraded":"2024-04-26T20:09:13.271089+0000","last_fullsized":"2024-04-26T20:09:13.271089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:05.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.280503+0000","last_change":"2024-04-26T20:08:25.714554+0000","last_active":"2024-04-26T20:09:13.280503+0000","last_peered":"2024-04-26T20:09:13.280503+0000","last_clean":"2024-04-26T20:09:13.280503+0000","last_became_active":"2024-04-26T20:08:25.711975+0000","last_became_peered":"2024-04-26T20:08:25.711975+0000","last_unstale":"2024-04-26T20:09:13.280503+0000","last_undegraded":"2024-04-26T20:09:13.280503+0000","last_fullsized":"2024-04-26T20:09:13.280503+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:20.061288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.346379+0000","last_change":"2024-04-26T20:08:15.634841+0000","last_active":"2024-04-26T20:09:22.346379+0000","last_peered":"2024-04-26T20:09:22.346379+0000","last_clean":"2024-04-26T20:09:22.346379+0000","last_became_active":"2024-04-26T20:08:15.633138+0000","last_became_peered":"2024-04-26T20:08:15.633138+0000","last_unstale":"2024-04-26T20:09:22.346379+0000","last_undegraded":"2024-04-26T20:09:22.346379+0000","last_fullsized":"2024-04-26T20:09:22.346379+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:19.364982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311398+0000","last_change":"2024-04-26T20:08:17.650201+0000","last_active":"2024-04-26T20:09:13.311398+0000","last_peered":"2024-04-26T20:09:13.311398+0000","last_clean":"2024-04-26T20:09:13.311398+0000","last_became_active":"2024-04-26T20:08:17.649819+0000","last_became_peered":"2024-04-26T20:08:17.649819+0000","last_unstale":"2024-04-26T20:09:13.311398+0000","last_undegraded":"2024-04-26T20:09:13.311398+0000","last_fullsized":"2024-04-26T20:09:13.311398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:01.772268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"94'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.316245+0000","last_change":"2024-04-26T20:09:13.315540+0000","last_active":"2024-04-26T20:09:13.316245+0000","last_peered":"2024-04-26T20:09:13.316245+0000","last_clean":"2024-04-26T20:09:13.316245+0000","last_became_active":"2024-04-26T20:09:13.314599+0000","last_became_peered":"2024-04-26T20:09:13.314599+0000","last_unstale":"2024-04-26T20:09:13.316245+0000","last_undegraded":"2024-04-26T20:09:13.316245+0000","last_fullsized":"2024-04-26T20:09:13.316245+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:29.926505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268205+0000","last_change":"2024-04-26T20:08:15.625290+0000","last_active":"2024-04-26T20:09:13.268205+0000","last_peered":"2024-04-26T20:09:13.268205+0000","last_clean":"2024-04-26T20:09:13.268205+0000","last_became_active":"2024-04-26T20:08:15.624310+0000","last_became_peered":"2024-04-26T20:08:15.624310+0000","last_unstale":"2024-04-26T20:09:13.268205+0000","last_undegraded":"2024-04-26T20:09:13.268205+0000","last_fullsized":"2024-04-26T20:09:13.268205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:04:16.736552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.314908+0000","last_change":"2024-04-26T20:08:17.665648+0000","last_active":"2024-04-26T20:09:13.314908+0000","last_peered":"2024-04-26T20:09:13.314908+0000","last_clean":"2024-04-26T20:09:13.314908+0000","last_became_active":"2024-04-26T20:08:17.665205+0000","last_became_peered":"2024-04-26T20:08:17.665205+0000","last_unstale":"2024-04-26T20:09:13.314908+0000","last_undegraded":"2024-04-26T20:09:13.314908+0000","last_fullsized":"2024-04-26T20:09:13.314908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:51.138181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270979+0000","last_change":"2024-04-26T20:08:25.708815+0000","last_active":"2024-04-26T20:09:13.270979+0000","last_peered":"2024-04-26T20:09:13.270979+0000","last_clean":"2024-04-26T20:09:13.270979+0000","last_became_active":"2024-04-26T20:08:25.708332+0000","last_became_peered":"2024-04-26T20:08:25.708332+0000","last_unstale":"2024-04-26T20:09:13.270979+0000","last_undegraded":"2024-04-26T20:09:13.270979+0000","last_fullsized":"2024-04-26T20:09:13.270979+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:23:19.378956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"94'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.344165+0000","last_change":"2024-04-26T20:08:15.626565+0000","last_active":"2024-04-26T20:09:22.344165+0000","last_peered":"2024-04-26T20:09:22.344165+0000","last_clean":"2024-04-26T20:09:22.344165+0000","last_became_active":"2024-04-26T20:08:15.626193+0000","last_became_peered":"2024-04-26T20:08:15.626193+0000","last_unstale":"2024-04-26T20:09:22.344165+0000","last_undegraded":"2024-04-26T20:09:22.344165+0000","last_fullsized":"2024-04-26T20:09:22.344165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:03:07.382550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313506+0000","last_change":"2024-04-26T20:08:17.658650+0000","last_active":"2024-04-26T20:09:13.313506+0000","last_peered":"2024-04-26T20:09:13.313506+0000","last_clean":"2024-04-26T20:09:13.313506+0000","last_became_active":"2024-04-26T20:08:17.658325+0000","last_became_peered":"2024-04-26T20:08:17.658325+0000","last_unstale":"2024-04-26T20:09:13.313506+0000","last_undegraded":"2024-04-26T20:09:13.313506+0000","last_fullsized":"2024-04-26T20:09:13.313506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:50:27.167289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312351+0000","last_change":"2024-04-26T20:08:25.681040+0000","last_active":"2024-04-26T20:09:13.312351+0000","last_peered":"2024-04-26T20:09:13.312351+0000","last_clean":"2024-04-26T20:09:13.312351+0000","last_became_active":"2024-04-26T20:08:25.680632+0000","last_became_peered":"2024-04-26T20:08:25.680632+0000","last_unstale":"2024-04-26T20:09:13.312351+0000","last_undegraded":"2024-04-26T20:09:13.312351+0000","last_fullsized":"2024-04-26T20:09:13.312351+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:16.017414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.361951+0000","last_change":"2024-04-26T20:08:16.123274+0000","last_active":"2024-04-26T20:09:23.361951+0000","last_peered":"2024-04-26T20:09:23.361951+0000","last_clean":"2024-04-26T20:09:23.361951+0000","last_became_active":"2024-04-26T20:08:16.122804+0000","last_became_peered":"2024-04-26T20:08:16.122804+0000","last_unstale":"2024-04-26T20:09:23.361951+0000","last_undegraded":"2024-04-26T20:09:23.361951+0000","last_fullsized":"2024-04-26T20:09:23.361951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:25.235296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313283+0000","last_change":"2024-04-26T20:08:17.641023+0000","last_active":"2024-04-26T20:09:13.313283+0000","last_peered":"2024-04-26T20:09:13.313283+0000","last_clean":"2024-04-26T20:09:13.313283+0000","last_became_active":"2024-04-26T20:08:17.640679+0000","last_became_peered":"2024-04-26T20:08:17.640679+0000","last_unstale":"2024-04-26T20:09:13.313283+0000","last_undegraded":"2024-04-26T20:09:13.313283+0000","last_fullsized":"2024-04-26T20:09:13.313283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:16:37.898958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269339+0000","last_change":"2024-04-26T20:08:25.704251+0000","last_active":"2024-04-26T20:09:13.269339+0000","last_peered":"2024-04-26T20:09:13.269339+0000","last_clean":"2024-04-26T20:09:13.269339+0000","last_became_active":"2024-04-26T20:08:25.702249+0000","last_became_peered":"2024-04-26T20:08:25.702249+0000","last_unstale":"2024-04-26T20:09:13.269339+0000","last_undegraded":"2024-04-26T20:09:13.269339+0000","last_fullsized":"2024-04-26T20:09:13.269339+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:03:12.107793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272597+0000","last_change":"2024-04-26T20:08:15.636474+0000","last_active":"2024-04-26T20:09:13.272597+0000","last_peered":"2024-04-26T20:09:13.272597+0000","last_clean":"2024-04-26T20:09:13.272597+0000","last_became_active":"2024-04-26T20:08:15.636104+0000","last_became_peered":"2024-04-26T20:08:15.636104+0000","last_unstale":"2024-04-26T20:09:13.272597+0000","last_undegraded":"2024-04-26T20:09:13.272597+0000","last_fullsized":"2024-04-26T20:09:13.272597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:09.514033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312629+0000","last_change":"2024-04-26T20:08:17.665683+0000","last_active":"2024-04-26T20:09:13.312629+0000","last_peered":"2024-04-26T20:09:13.312629+0000","last_clean":"2024-04-26T20:09:13.312629+0000","last_became_active":"2024-04-26T20:08:17.665385+0000","last_became_peered":"2024-04-26T20:08:17.665385+0000","last_unstale":"2024-04-26T20:09:13.312629+0000","last_undegraded":"2024-04-26T20:09:13.312629+0000","last_fullsized":"2024-04-26T20:09:13.312629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:00.826582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271549+0000","last_change":"2024-04-26T20:08:25.683715+0000","last_active":"2024-04-26T20:09:13.271549+0000","last_peered":"2024-04-26T20:09:13.271549+0000","last_clean":"2024-04-26T20:09:13.271549+0000","last_became_active":"2024-04-26T20:08:25.683155+0000","last_became_peered":"2024-04-26T20:08:25.683155+0000","last_unstale":"2024-04-26T20:09:13.271549+0000","last_undegraded":"2024-04-26T20:09:13.271549+0000","last_fullsized":"2024-04-26T20:09:13.271549+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:44.751519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.278720+0000","last_change":"2024-04-26T20:08:15.632519+0000","last_active":"2024-04-26T20:09:13.278720+0000","last_peered":"2024-04-26T20:09:13.278720+0000","last_clean":"2024-04-26T20:09:13.278720+0000","last_became_active":"2024-04-26T20:08:15.631976+0000","last_became_peered":"2024-04-26T20:08:15.631976+0000","last_unstale":"2024-04-26T20:09:13.278720+0000","last_undegraded":"2024-04-26T20:09:13.278720+0000","last_fullsized":"2024-04-26T20:09:13.278720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:39:57.668441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313301+0000","last_change":"2024-04-26T20:08:17.642480+0000","last_active":"2024-04-26T20:09:13.313301+0000","last_peered":"2024-04-26T20:09:13.313301+0000","last_clean":"2024-04-26T20:09:13.313301+0000","last_became_active":"2024-04-26T20:08:17.642165+0000","last_became_peered":"2024-04-26T20:08:17.642165+0000","last_unstale":"2024-04-26T20:09:13.313301+0000","last_undegraded":"2024-04-26T20:09:13.313301+0000","last_fullsized":"2024-04-26T20:09:13.313301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:05:39.440740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311443+0000","last_change":"2024-04-26T20:08:17.645823+0000","last_active":"2024-04-26T20:09:13.311443+0000","last_peered":"2024-04-26T20:09:13.311443+0000","last_clean":"2024-04-26T20:09:13.311443+0000","last_became_active":"2024-04-26T20:08:17.645413+0000","last_became_peered":"2024-04-26T20:08:17.645413+0000","last_unstale":"2024-04-26T20:09:13.311443+0000","last_undegraded":"2024-04-26T20:09:13.311443+0000","last_fullsized":"2024-04-26T20:09:13.311443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:36:36.622579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268290+0000","last_change":"2024-04-26T20:08:17.641413+0000","last_active":"2024-04-26T20:09:13.268290+0000","last_peered":"2024-04-26T20:09:13.268290+0000","last_clean":"2024-04-26T20:09:13.268290+0000","last_became_active":"2024-04-26T20:08:17.641073+0000","last_became_peered":"2024-04-26T20:08:17.641073+0000","last_unstale":"2024-04-26T20:09:13.268290+0000","last_undegraded":"2024-04-26T20:09:13.268290+0000","last_fullsized":"2024-04-26T20:09:13.268290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:25:03.961399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268049+0000","last_change":"2024-04-26T20:08:17.657336+0000","last_active":"2024-04-26T20:09:13.268049+0000","last_peered":"2024-04-26T20:09:13.268049+0000","last_clean":"2024-04-26T20:09:13.268049+0000","last_became_active":"2024-04-26T20:08:17.656982+0000","last_became_peered":"2024-04-26T20:08:17.656982+0000","last_unstale":"2024-04-26T20:09:13.268049+0000","last_undegraded":"2024-04-26T20:09:13.268049+0000","last_fullsized":"2024-04-26T20:09:13.268049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:10.149705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269974+0000","last_change":"2024-04-26T20:08:17.662290+0000","last_active":"2024-04-26T20:09:13.269974+0000","last_peered":"2024-04-26T20:09:13.269974+0000","last_clean":"2024-04-26T20:09:13.269974+0000","last_became_active":"2024-04-26T20:08:17.660429+0000","last_became_peered":"2024-04-26T20:08:17.660429+0000","last_unstale":"2024-04-26T20:09:13.269974+0000","last_undegraded":"2024-04-26T20:09:13.269974+0000","last_fullsized":"2024-04-26T20:09:13.269974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:50:03.563066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.284038+0000","last_change":"2024-04-26T20:09:13.284038+0000","last_active":"2024-04-26T20:09:13.284038+0000","last_peered":"2024-04-26T20:09:13.284038+0000","last_clean":"2024-04-26T20:09:13.284038+0000","last_became_active":"2024-04-26T20:09:13.283650+0000","last_became_peered":"2024-04-26T20:09:13.283650+0000","last_unstale":"2024-04-26T20:09:13.284038+0000","last_undegraded":"2024-04-26T20:09:13.284038+0000","last_fullsized":"2024-04-26T20:09:13.284038+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:33.570782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311739+0000","last_change":"2024-04-26T20:08:17.642813+0000","last_active":"2024-04-26T20:09:13.311739+0000","last_peered":"2024-04-26T20:09:13.311739+0000","last_clean":"2024-04-26T20:09:13.311739+0000","last_became_active":"2024-04-26T20:08:17.642018+0000","last_became_peered":"2024-04-26T20:08:17.642018+0000","last_unstale":"2024-04-26T20:09:13.311739+0000","last_undegraded":"2024-04-26T20:09:13.311739+0000","last_fullsized":"2024-04-26T20:09:13.311739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:19.029810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268589+0000","last_change":"2024-04-26T20:08:17.639030+0000","last_active":"2024-04-26T20:09:13.268589+0000","last_peered":"2024-04-26T20:09:13.268589+0000","last_clean":"2024-04-26T20:09:13.268589+0000","last_became_active":"2024-04-26T20:08:17.638645+0000","last_became_peered":"2024-04-26T20:08:17.638645+0000","last_unstale":"2024-04-26T20:09:13.268589+0000","last_undegraded":"2024-04-26T20:09:13.268589+0000","last_fullsized":"2024-04-26T20:09:13.268589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:58.373648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.316511+0000","last_change":"2024-04-26T20:08:17.666873+0000","last_active":"2024-04-26T20:09:13.316511+0000","last_peered":"2024-04-26T20:09:13.316511+0000","last_clean":"2024-04-26T20:09:13.316511+0000","last_became_active":"2024-04-26T20:08:17.666185+0000","last_became_peered":"2024-04-26T20:08:17.666185+0000","last_unstale":"2024-04-26T20:09:13.316511+0000","last_undegraded":"2024-04-26T20:09:13.316511+0000","last_fullsized":"2024-04-26T20:09:13.316511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:51:17.105637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271849+0000","last_change":"2024-04-26T20:08:17.666625+0000","last_active":"2024-04-26T20:09:13.271849+0000","last_peered":"2024-04-26T20:09:13.271849+0000","last_clean":"2024-04-26T20:09:13.271849+0000","last_became_active":"2024-04-26T20:08:17.666288+0000","last_became_peered":"2024-04-26T20:08:17.666288+0000","last_unstale":"2024-04-26T20:09:13.271849+0000","last_undegraded":"2024-04-26T20:09:13.271849+0000","last_fullsized":"2024-04-26T20:09:13.271849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:27.051752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.315490+0000","last_change":"2024-04-26T20:08:17.663490+0000","last_active":"2024-04-26T20:09:13.315490+0000","last_peered":"2024-04-26T20:09:13.315490+0000","last_clean":"2024-04-26T20:09:13.315490+0000","last_became_active":"2024-04-26T20:08:17.663131+0000","last_became_peered":"2024-04-26T20:08:17.663131+0000","last_unstale":"2024-04-26T20:09:13.315490+0000","last_undegraded":"2024-04-26T20:09:13.315490+0000","last_fullsized":"2024-04-26T20:09:13.315490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:46.633558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312640+0000","last_change":"2024-04-26T20:08:17.664749+0000","last_active":"2024-04-26T20:09:13.312640+0000","last_peered":"2024-04-26T20:09:13.312640+0000","last_clean":"2024-04-26T20:09:13.312640+0000","last_became_active":"2024-04-26T20:08:17.664346+0000","last_became_peered":"2024-04-26T20:08:17.664346+0000","last_unstale":"2024-04-26T20:09:13.312640+0000","last_undegraded":"2024-04-26T20:09:13.312640+0000","last_fullsized":"2024-04-26T20:09:13.312640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:27.106743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279548+0000","last_change":"2024-04-26T20:08:17.662955+0000","last_active":"2024-04-26T20:09:13.279548+0000","last_peered":"2024-04-26T20:09:13.279548+0000","last_clean":"2024-04-26T20:09:13.279548+0000","last_became_active":"2024-04-26T20:08:17.662639+0000","last_became_peered":"2024-04-26T20:08:17.662639+0000","last_unstale":"2024-04-26T20:09:13.279548+0000","last_undegraded":"2024-04-26T20:09:13.279548+0000","last_fullsized":"2024-04-26T20:09:13.279548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:28.346696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311667+0000","last_change":"2024-04-26T20:08:17.641465+0000","last_active":"2024-04-26T20:09:13.311667+0000","last_peered":"2024-04-26T20:09:13.311667+0000","last_clean":"2024-04-26T20:09:13.311667+0000","last_became_active":"2024-04-26T20:08:17.641145+0000","last_became_peered":"2024-04-26T20:08:17.641145+0000","last_unstale":"2024-04-26T20:09:13.311667+0000","last_undegraded":"2024-04-26T20:09:13.311667+0000","last_fullsized":"2024-04-26T20:09:13.311667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:58.977174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":47342,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":156,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":150580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":525894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28716,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716436,"statfs":{"total":95995035648,"available":95965630464,"internally_reserved":0,"allocated":1740800,"data_stored":1160599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":526130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070512,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":526050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1121959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":521306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":564557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1085440,"data_stored":522694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":574852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":521500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":525956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1683456,"data_stored":1117547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":43341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":44919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T20:09:41.245 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T20:09:41.255 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:40 smithi094 ceph-mon[28050]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:09:41.263 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:40 smithi204 ceph-mon[32618]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:09:41.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:40 smithi103 ceph-mon[32635]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:09:41.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.477+0000 7fc0ea600640 1 -- 172.21.15.94:0/2724596371 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4108d90 msgr2=0x7fc0e41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:41.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.477+0000 7fc0ea600640 1 --2- 172.21.15.94:0/2724596371 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4108d90 0x7fc0e41091b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009900 tx=0x7fc0d402a410 comp rx=0 tx=0).stop 2024-04-26T20:09:41.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.477+0000 7fc0ea600640 1 -- 172.21.15.94:0/2724596371 shutdown_connections 2024-04-26T20:09:41.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.477+0000 7fc0ea600640 1 --2- 172.21.15.94:0/2724596371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0e410a940 0x7fc0e41111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:41.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.477+0000 7fc0ea600640 1 --2- 172.21.15.94:0/2724596371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0e4109f80 0x7fc0e410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:41.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.477+0000 7fc0ea600640 1 --2- 172.21.15.94:0/2724596371 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4108d90 0x7fc0e41091b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:41.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.477+0000 7fc0ea600640 1 -- 172.21.15.94:0/2724596371 >> 172.21.15.94:0/2724596371 conn(0x7fc0e4077d30 msgr2=0x7fc0e407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:41.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.477+0000 7fc0ea600640 1 -- 172.21.15.94:0/2724596371 shutdown_connections 2024-04-26T20:09:41.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.477+0000 7fc0ea600640 1 -- 172.21.15.94:0/2724596371 wait complete. 2024-04-26T20:09:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.478+0000 7fc0ea600640 1 Processor -- start 2024-04-26T20:09:41.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0ea600640 1 -- start start 2024-04-26T20:09:41.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0e4108d90 0x7fc0e419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:41.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4109f80 0x7fc0e419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:41.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0e410a940 0x7fc0e41a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:41.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0ea600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e41148e0 con 0x7fc0e4109f80 2024-04-26T20:09:41.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0ea600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e4114760 con 0x7fc0e410a940 2024-04-26T20:09:41.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0ea600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e4114a60 con 0x7fc0e4108d90 2024-04-26T20:09:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0dbe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4109f80 0x7fc0e419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0dbe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4109f80 0x7fc0e419f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51682/0 (socket says 172.21.15.94:51682) 2024-04-26T20:09:41.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0dbe00640 1 -- 172.21.15.94:0/3160314315 learned_addr learned my addr 172.21.15.94:0/3160314315 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:41.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0e9200640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0e4108d90 0x7fc0e419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:41.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.479+0000 7fc0e9c00640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0e410a940 0x7fc0e41a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:41.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0dbe00640 1 -- 172.21.15.94:0/3160314315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0e4108d90 msgr2=0x7fc0e419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:41.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0dbe00640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0e4108d90 0x7fc0e419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:41.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0dbe00640 1 -- 172.21.15.94:0/3160314315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0e410a940 msgr2=0x7fc0e41a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:41.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0dbe00640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0e410a940 0x7fc0e41a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:41.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0dbe00640 1 -- 172.21.15.94:0/3160314315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e41a6bd0 con 0x7fc0e4109f80 2024-04-26T20:09:41.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0e9200640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0e4108d90 0x7fc0e419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:09:41.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0dbe00640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4109f80 0x7fc0e419f4f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00e9f0 tx=0x7fc0cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:41.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0e2000640 1 -- 172.21.15.94:0/3160314315 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc00cd80 con 0x7fc0e4109f80 2024-04-26T20:09:41.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0e2000640 1 -- 172.21.15.94:0/3160314315 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0cc0108e0 con 0x7fc0e4109f80 2024-04-26T20:09:41.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0e2000640 1 -- 172.21.15.94:0/3160314315 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc018980 con 0x7fc0e4109f80 2024-04-26T20:09:41.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e41a6ec0 con 0x7fc0e4109f80 2024-04-26T20:09:41.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.480+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e41a73d0 con 0x7fc0e4109f80 2024-04-26T20:09:42.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.481+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e4110180 con 0x7fc0e4109f80 2024-04-26T20:09:42.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.483+0000 7fc0e2000640 1 -- 172.21.15.94:0/3160314315 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc0cc010430 con 0x7fc0e4109f80 2024-04-26T20:09:42.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.485+0000 7fc0e2000640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc0c80a3cc0 0x7fc0c80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:42.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.485+0000 7fc0e2000640 1 -- 172.21.15.94:0/3160314315 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fc0cc0d0c60 con 0x7fc0e4109f80 2024-04-26T20:09:42.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.485+0000 7fc0e9200640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc0c80a3cc0 0x7fc0c80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:42.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.486+0000 7fc0e9200640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc0c80a3cc0 0x7fc0c80a6180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006010 tx=0x7fc0d40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:42.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.489+0000 7fc0e2000640 1 -- 172.21.15.94:0/3160314315 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc0cc09d120 con 0x7fc0e4109f80 2024-04-26T20:09:42.146 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:41 smithi204 ceph-mon[32618]: pgmap v369: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-04-26T20:09:42.153 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:41 smithi204 ceph-mon[32618]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:09:42.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.692+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc0e4062d70 con 0x7fc0c80a3cc0 2024-04-26T20:09:42.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.703+0000 7fc0e2000640 1 -- 172.21.15.94:0/3160314315 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340357 (secure 0 0 0) 0x7fc0e4062d70 con 0x7fc0c80a3cc0 2024-04-26T20:09:42.245 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:42.254 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-26T20:09:42.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc0c80a3cc0 msgr2=0x7fc0c80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:42.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc0c80a3cc0 0x7fc0c80a6180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006010 tx=0x7fc0d40314a0 comp rx=0 tx=0).stop 2024-04-26T20:09:42.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4109f80 msgr2=0x7fc0e419f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:42.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4109f80 0x7fc0e419f4f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00e9f0 tx=0x7fc0cc00eeb0 comp rx=0 tx=0).stop 2024-04-26T20:09:42.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 shutdown_connections 2024-04-26T20:09:42.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc0c80a3cc0 0x7fc0c80a6180 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:42.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0e410a940 0x7fc0e41a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0e4109f80 0x7fc0e419f4f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:42.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 --2- 172.21.15.94:0/3160314315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0e4108d90 0x7fc0e419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:42.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 >> 172.21.15.94:0/3160314315 conn(0x7fc0e4077d30 msgr2=0x7fc0e410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:42.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 shutdown_connections 2024-04-26T20:09:42.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:41.710+0000 7fc0ea600640 1 -- 172.21.15.94:0/3160314315 wait complete. 2024-04-26T20:09:42.456 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-04-26T20:09:41.592187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646498,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":241,"num_read_kb":227,"num_write":343,"num_write_kb":3045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392812,"kb_used_data":14540,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549012,"statfs":{"total":1151940427776,"available":1151538188288,"internally_reserved":0,"allocated":14888960,"data_stored":8226152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":387233084},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":48678,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":90,"num_write":32,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005869"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311966+0000","last_change":"2024-04-26T20:08:17.653922+0000","last_active":"2024-04-26T20:09:13.311966+0000","last_peered":"2024-04-26T20:09:13.311966+0000","last_clean":"2024-04-26T20:09:13.311966+0000","last_became_active":"2024-04-26T20:08:17.653486+0000","last_became_peered":"2024-04-26T20:08:17.653486+0000","last_unstale":"2024-04-26T20:09:13.311966+0000","last_undegraded":"2024-04-26T20:09:13.311966+0000","last_fullsized":"2024-04-26T20:09:13.311966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:05.034829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269659+0000","last_change":"2024-04-26T20:08:17.650126+0000","last_active":"2024-04-26T20:09:13.269659+0000","last_peered":"2024-04-26T20:09:13.269659+0000","last_clean":"2024-04-26T20:09:13.269659+0000","last_became_active":"2024-04-26T20:08:17.649727+0000","last_became_peered":"2024-04-26T20:08:17.649727+0000","last_unstale":"2024-04-26T20:09:13.269659+0000","last_undegraded":"2024-04-26T20:09:13.269659+0000","last_fullsized":"2024-04-26T20:09:13.269659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:11:28.255286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270403+0000","last_change":"2024-04-26T20:08:17.646243+0000","last_active":"2024-04-26T20:09:13.270403+0000","last_peered":"2024-04-26T20:09:13.270403+0000","last_clean":"2024-04-26T20:09:13.270403+0000","last_became_active":"2024-04-26T20:08:17.645677+0000","last_became_peered":"2024-04-26T20:08:17.645677+0000","last_unstale":"2024-04-26T20:09:13.270403+0000","last_undegraded":"2024-04-26T20:09:13.270403+0000","last_fullsized":"2024-04-26T20:09:13.270403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:31.698329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272081+0000","last_change":"2024-04-26T20:08:17.639845+0000","last_active":"2024-04-26T20:09:13.272081+0000","last_peered":"2024-04-26T20:09:13.272081+0000","last_clean":"2024-04-26T20:09:13.272081+0000","last_became_active":"2024-04-26T20:08:17.639465+0000","last_became_peered":"2024-04-26T20:08:17.639465+0000","last_unstale":"2024-04-26T20:09:13.272081+0000","last_undegraded":"2024-04-26T20:09:13.272081+0000","last_fullsized":"2024-04-26T20:09:13.272081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:05.717196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271298+0000","last_change":"2024-04-26T20:08:17.660459+0000","last_active":"2024-04-26T20:09:13.271298+0000","last_peered":"2024-04-26T20:09:13.271298+0000","last_clean":"2024-04-26T20:09:13.271298+0000","last_became_active":"2024-04-26T20:08:17.659927+0000","last_became_peered":"2024-04-26T20:08:17.659927+0000","last_unstale":"2024-04-26T20:09:13.271298+0000","last_undegraded":"2024-04-26T20:09:13.271298+0000","last_fullsized":"2024-04-26T20:09:13.271298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:10.062073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279825+0000","last_change":"2024-04-26T20:09:13.279825+0000","last_active":"2024-04-26T20:09:13.279825+0000","last_peered":"2024-04-26T20:09:13.279825+0000","last_clean":"2024-04-26T20:09:13.279825+0000","last_became_active":"2024-04-26T20:09:13.279464+0000","last_became_peered":"2024-04-26T20:09:13.279464+0000","last_unstale":"2024-04-26T20:09:13.279825+0000","last_undegraded":"2024-04-26T20:09:13.279825+0000","last_fullsized":"2024-04-26T20:09:13.279825+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:41.687987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312677+0000","last_change":"2024-04-26T20:08:17.648806+0000","last_active":"2024-04-26T20:09:13.312677+0000","last_peered":"2024-04-26T20:09:13.312677+0000","last_clean":"2024-04-26T20:09:13.312677+0000","last_became_active":"2024-04-26T20:08:17.648552+0000","last_became_peered":"2024-04-26T20:08:17.648552+0000","last_unstale":"2024-04-26T20:09:13.312677+0000","last_undegraded":"2024-04-26T20:09:13.312677+0000","last_fullsized":"2024-04-26T20:09:13.312677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:43.282610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313087+0000","last_change":"2024-04-26T20:08:17.663479+0000","last_active":"2024-04-26T20:09:13.313087+0000","last_peered":"2024-04-26T20:09:13.313087+0000","last_clean":"2024-04-26T20:09:13.313087+0000","last_became_active":"2024-04-26T20:08:17.663043+0000","last_became_peered":"2024-04-26T20:08:17.663043+0000","last_unstale":"2024-04-26T20:09:13.313087+0000","last_undegraded":"2024-04-26T20:09:13.313087+0000","last_fullsized":"2024-04-26T20:09:13.313087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:21:36.518513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313828+0000","last_change":"2024-04-26T20:08:17.637978+0000","last_active":"2024-04-26T20:09:13.313828+0000","last_peered":"2024-04-26T20:09:13.313828+0000","last_clean":"2024-04-26T20:09:13.313828+0000","last_became_active":"2024-04-26T20:08:17.637458+0000","last_became_peered":"2024-04-26T20:08:17.637458+0000","last_unstale":"2024-04-26T20:09:13.313828+0000","last_undegraded":"2024-04-26T20:09:13.313828+0000","last_fullsized":"2024-04-26T20:09:13.313828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:20:01.232741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.274941+0000","last_change":"2024-04-26T20:08:17.639819+0000","last_active":"2024-04-26T20:09:13.274941+0000","last_peered":"2024-04-26T20:09:13.274941+0000","last_clean":"2024-04-26T20:09:13.274941+0000","last_became_active":"2024-04-26T20:08:17.639479+0000","last_became_peered":"2024-04-26T20:08:17.639479+0000","last_unstale":"2024-04-26T20:09:13.274941+0000","last_undegraded":"2024-04-26T20:09:13.274941+0000","last_fullsized":"2024-04-26T20:09:13.274941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:44.536585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313378+0000","last_change":"2024-04-26T20:08:17.642533+0000","last_active":"2024-04-26T20:09:13.313378+0000","last_peered":"2024-04-26T20:09:13.313378+0000","last_clean":"2024-04-26T20:09:13.313378+0000","last_became_active":"2024-04-26T20:08:17.642198+0000","last_became_peered":"2024-04-26T20:08:17.642198+0000","last_unstale":"2024-04-26T20:09:13.313378+0000","last_undegraded":"2024-04-26T20:09:13.313378+0000","last_fullsized":"2024-04-26T20:09:13.313378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:16:21.200183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271602+0000","last_change":"2024-04-26T20:08:17.644646+0000","last_active":"2024-04-26T20:09:13.271602+0000","last_peered":"2024-04-26T20:09:13.271602+0000","last_clean":"2024-04-26T20:09:13.271602+0000","last_became_active":"2024-04-26T20:08:17.644314+0000","last_became_peered":"2024-04-26T20:08:17.644314+0000","last_unstale":"2024-04-26T20:09:13.271602+0000","last_undegraded":"2024-04-26T20:09:13.271602+0000","last_fullsized":"2024-04-26T20:09:13.271602+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:55:00.003120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279260+0000","last_change":"2024-04-26T20:08:17.657118+0000","last_active":"2024-04-26T20:09:13.279260+0000","last_peered":"2024-04-26T20:09:13.279260+0000","last_clean":"2024-04-26T20:09:13.279260+0000","last_became_active":"2024-04-26T20:08:17.656256+0000","last_became_peered":"2024-04-26T20:08:17.656256+0000","last_unstale":"2024-04-26T20:09:13.279260+0000","last_undegraded":"2024-04-26T20:09:13.279260+0000","last_fullsized":"2024-04-26T20:09:13.279260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:09.133475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269622+0000","last_change":"2024-04-26T20:08:17.658234+0000","last_active":"2024-04-26T20:09:13.269622+0000","last_peered":"2024-04-26T20:09:13.269622+0000","last_clean":"2024-04-26T20:09:13.269622+0000","last_became_active":"2024-04-26T20:08:17.651714+0000","last_became_peered":"2024-04-26T20:08:17.651714+0000","last_unstale":"2024-04-26T20:09:13.269622+0000","last_undegraded":"2024-04-26T20:09:13.269622+0000","last_fullsized":"2024-04-26T20:09:13.269622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:30.630320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269216+0000","last_change":"2024-04-26T20:08:17.647898+0000","last_active":"2024-04-26T20:09:13.269216+0000","last_peered":"2024-04-26T20:09:13.269216+0000","last_clean":"2024-04-26T20:09:13.269216+0000","last_became_active":"2024-04-26T20:08:17.647463+0000","last_became_peered":"2024-04-26T20:08:17.647463+0000","last_unstale":"2024-04-26T20:09:13.269216+0000","last_undegraded":"2024-04-26T20:09:13.269216+0000","last_fullsized":"2024-04-26T20:09:13.269216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:24.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311681+0000","last_change":"2024-04-26T20:08:17.639518+0000","last_active":"2024-04-26T20:09:13.311681+0000","last_peered":"2024-04-26T20:09:13.311681+0000","last_clean":"2024-04-26T20:09:13.311681+0000","last_became_active":"2024-04-26T20:08:17.639183+0000","last_became_peered":"2024-04-26T20:08:17.639183+0000","last_unstale":"2024-04-26T20:09:13.311681+0000","last_undegraded":"2024-04-26T20:09:13.311681+0000","last_fullsized":"2024-04-26T20:09:13.311681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:41:12.893323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311825+0000","last_change":"2024-04-26T20:08:17.653956+0000","last_active":"2024-04-26T20:09:13.311825+0000","last_peered":"2024-04-26T20:09:13.311825+0000","last_clean":"2024-04-26T20:09:13.311825+0000","last_became_active":"2024-04-26T20:08:17.653576+0000","last_became_peered":"2024-04-26T20:08:17.653576+0000","last_unstale":"2024-04-26T20:09:13.311825+0000","last_undegraded":"2024-04-26T20:09:13.311825+0000","last_fullsized":"2024-04-26T20:09:13.311825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:35:09.538194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.281786+0000","last_change":"2024-04-26T20:08:17.649500+0000","last_active":"2024-04-26T20:09:13.281786+0000","last_peered":"2024-04-26T20:09:13.281786+0000","last_clean":"2024-04-26T20:09:13.281786+0000","last_became_active":"2024-04-26T20:08:17.649062+0000","last_became_peered":"2024-04-26T20:08:17.649062+0000","last_unstale":"2024-04-26T20:09:13.281786+0000","last_undegraded":"2024-04-26T20:09:13.281786+0000","last_fullsized":"2024-04-26T20:09:13.281786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:24.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268715+0000","last_change":"2024-04-26T20:08:17.666440+0000","last_active":"2024-04-26T20:09:13.268715+0000","last_peered":"2024-04-26T20:09:13.268715+0000","last_clean":"2024-04-26T20:09:13.268715+0000","last_became_active":"2024-04-26T20:08:17.666024+0000","last_became_peered":"2024-04-26T20:08:17.666024+0000","last_unstale":"2024-04-26T20:09:13.268715+0000","last_undegraded":"2024-04-26T20:09:13.268715+0000","last_fullsized":"2024-04-26T20:09:13.268715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:55.356424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268092+0000","last_change":"2024-04-26T20:08:25.700728+0000","last_active":"2024-04-26T20:09:13.268092+0000","last_peered":"2024-04-26T20:09:13.268092+0000","last_clean":"2024-04-26T20:09:13.268092+0000","last_became_active":"2024-04-26T20:08:25.700339+0000","last_became_peered":"2024-04-26T20:08:25.700339+0000","last_unstale":"2024-04-26T20:09:13.268092+0000","last_undegraded":"2024-04-26T20:09:13.268092+0000","last_fullsized":"2024-04-26T20:09:13.268092+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:53.581083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.363260+0000","last_change":"2024-04-26T20:08:15.632803+0000","last_active":"2024-04-26T20:09:23.363260+0000","last_peered":"2024-04-26T20:09:23.363260+0000","last_clean":"2024-04-26T20:09:23.363260+0000","last_became_active":"2024-04-26T20:08:15.632368+0000","last_became_peered":"2024-04-26T20:08:15.632368+0000","last_unstale":"2024-04-26T20:09:23.363260+0000","last_undegraded":"2024-04-26T20:09:23.363260+0000","last_fullsized":"2024-04-26T20:09:23.363260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:55:54.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.315743+0000","last_change":"2024-04-26T20:08:17.657637+0000","last_active":"2024-04-26T20:09:13.315743+0000","last_peered":"2024-04-26T20:09:13.315743+0000","last_clean":"2024-04-26T20:09:13.315743+0000","last_became_active":"2024-04-26T20:08:17.657290+0000","last_became_peered":"2024-04-26T20:08:17.657290+0000","last_unstale":"2024-04-26T20:09:13.315743+0000","last_undegraded":"2024-04-26T20:09:13.315743+0000","last_fullsized":"2024-04-26T20:09:13.315743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:25.270989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.318084+0000","last_change":"2024-04-26T20:09:13.318084+0000","last_active":"2024-04-26T20:09:13.318084+0000","last_peered":"2024-04-26T20:09:13.318084+0000","last_clean":"2024-04-26T20:09:13.318084+0000","last_became_active":"2024-04-26T20:09:13.317689+0000","last_became_peered":"2024-04-26T20:09:13.317689+0000","last_unstale":"2024-04-26T20:09:13.318084+0000","last_undegraded":"2024-04-26T20:09:13.318084+0000","last_fullsized":"2024-04-26T20:09:13.318084+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:48.182186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.316341+0000","last_change":"2024-04-26T20:08:16.128717+0000","last_active":"2024-04-26T20:09:13.316341+0000","last_peered":"2024-04-26T20:09:13.316341+0000","last_clean":"2024-04-26T20:09:13.316341+0000","last_became_active":"2024-04-26T20:08:16.127738+0000","last_became_peered":"2024-04-26T20:08:16.127738+0000","last_unstale":"2024-04-26T20:09:13.316341+0000","last_undegraded":"2024-04-26T20:09:13.316341+0000","last_fullsized":"2024-04-26T20:09:13.316341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:01:26.343914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270890+0000","last_change":"2024-04-26T20:08:17.659613+0000","last_active":"2024-04-26T20:09:13.270890+0000","last_peered":"2024-04-26T20:09:13.270890+0000","last_clean":"2024-04-26T20:09:13.270890+0000","last_became_active":"2024-04-26T20:08:17.658750+0000","last_became_peered":"2024-04-26T20:08:17.658750+0000","last_unstale":"2024-04-26T20:09:13.270890+0000","last_undegraded":"2024-04-26T20:09:13.270890+0000","last_fullsized":"2024-04-26T20:09:13.270890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:59:08.724656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271319+0000","last_change":"2024-04-26T20:08:25.709115+0000","last_active":"2024-04-26T20:09:13.271319+0000","last_peered":"2024-04-26T20:09:13.271319+0000","last_clean":"2024-04-26T20:09:13.271319+0000","last_became_active":"2024-04-26T20:08:25.708552+0000","last_became_peered":"2024-04-26T20:08:25.708552+0000","last_unstale":"2024-04-26T20:09:13.271319+0000","last_undegraded":"2024-04-26T20:09:13.271319+0000","last_fullsized":"2024-04-26T20:09:13.271319+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:20.735727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"94'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.353071+0000","last_change":"2024-04-26T20:08:15.624441+0000","last_active":"2024-04-26T20:09:23.353071+0000","last_peered":"2024-04-26T20:09:23.353071+0000","last_clean":"2024-04-26T20:09:23.353071+0000","last_became_active":"2024-04-26T20:08:15.623999+0000","last_became_peered":"2024-04-26T20:08:15.623999+0000","last_unstale":"2024-04-26T20:09:23.353071+0000","last_undegraded":"2024-04-26T20:09:23.353071+0000","last_fullsized":"2024-04-26T20:09:23.353071+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:34.276360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311954+0000","last_change":"2024-04-26T20:08:17.656088+0000","last_active":"2024-04-26T20:09:13.311954+0000","last_peered":"2024-04-26T20:09:13.311954+0000","last_clean":"2024-04-26T20:09:13.311954+0000","last_became_active":"2024-04-26T20:08:17.655736+0000","last_became_peered":"2024-04-26T20:08:17.655736+0000","last_unstale":"2024-04-26T20:09:13.311954+0000","last_undegraded":"2024-04-26T20:09:13.311954+0000","last_fullsized":"2024-04-26T20:09:13.311954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:29.187893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270667+0000","last_change":"2024-04-26T20:08:25.703015+0000","last_active":"2024-04-26T20:09:13.270667+0000","last_peered":"2024-04-26T20:09:13.270667+0000","last_clean":"2024-04-26T20:09:13.270667+0000","last_became_active":"2024-04-26T20:08:25.702699+0000","last_became_peered":"2024-04-26T20:08:25.702699+0000","last_unstale":"2024-04-26T20:09:13.270667+0000","last_undegraded":"2024-04-26T20:09:13.270667+0000","last_fullsized":"2024-04-26T20:09:13.270667+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:29:17.082803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271542+0000","last_change":"2024-04-26T20:08:15.624092+0000","last_active":"2024-04-26T20:09:13.271542+0000","last_peered":"2024-04-26T20:09:13.271542+0000","last_clean":"2024-04-26T20:09:13.271542+0000","last_became_active":"2024-04-26T20:08:15.623540+0000","last_became_peered":"2024-04-26T20:08:15.623540+0000","last_unstale":"2024-04-26T20:09:13.271542+0000","last_undegraded":"2024-04-26T20:09:13.271542+0000","last_fullsized":"2024-04-26T20:09:13.271542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:06:17.162854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313449+0000","last_change":"2024-04-26T20:09:13.313449+0000","last_active":"2024-04-26T20:09:13.313449+0000","last_peered":"2024-04-26T20:09:13.313449+0000","last_clean":"2024-04-26T20:09:13.313449+0000","last_became_active":"2024-04-26T20:09:13.313124+0000","last_became_peered":"2024-04-26T20:09:13.313124+0000","last_unstale":"2024-04-26T20:09:13.313449+0000","last_undegraded":"2024-04-26T20:09:13.313449+0000","last_fullsized":"2024-04-26T20:09:13.313449+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:10.447987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268370+0000","last_change":"2024-04-26T20:08:25.684973+0000","last_active":"2024-04-26T20:09:13.268370+0000","last_peered":"2024-04-26T20:09:13.268370+0000","last_clean":"2024-04-26T20:09:13.268370+0000","last_became_active":"2024-04-26T20:08:25.684636+0000","last_became_peered":"2024-04-26T20:08:25.684636+0000","last_unstale":"2024-04-26T20:09:13.268370+0000","last_undegraded":"2024-04-26T20:09:13.268370+0000","last_fullsized":"2024-04-26T20:09:13.268370+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:47:17.140765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:36.558738+0000","last_change":"2024-04-26T20:08:15.626792+0000","last_active":"2024-04-26T20:09:36.558738+0000","last_peered":"2024-04-26T20:09:36.558738+0000","last_clean":"2024-04-26T20:09:36.558738+0000","last_became_active":"2024-04-26T20:08:15.626453+0000","last_became_peered":"2024-04-26T20:08:15.626453+0000","last_unstale":"2024-04-26T20:09:36.558738+0000","last_undegraded":"2024-04-26T20:09:36.558738+0000","last_fullsized":"2024-04-26T20:09:36.558738+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:27.835127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271204+0000","last_change":"2024-04-26T20:08:17.660345+0000","last_active":"2024-04-26T20:09:13.271204+0000","last_peered":"2024-04-26T20:09:13.271204+0000","last_clean":"2024-04-26T20:09:13.271204+0000","last_became_active":"2024-04-26T20:08:17.659966+0000","last_became_peered":"2024-04-26T20:08:17.659966+0000","last_unstale":"2024-04-26T20:09:13.271204+0000","last_undegraded":"2024-04-26T20:09:13.271204+0000","last_fullsized":"2024-04-26T20:09:13.271204+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:48.762984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268348+0000","last_change":"2024-04-26T20:08:25.698002+0000","last_active":"2024-04-26T20:09:13.268348+0000","last_peered":"2024-04-26T20:09:13.268348+0000","last_clean":"2024-04-26T20:09:13.268348+0000","last_became_active":"2024-04-26T20:08:25.697588+0000","last_became_peered":"2024-04-26T20:08:25.697588+0000","last_unstale":"2024-04-26T20:09:13.268348+0000","last_undegraded":"2024-04-26T20:09:13.268348+0000","last_fullsized":"2024-04-26T20:09:13.268348+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:59.771941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.353673+0000","last_change":"2024-04-26T20:08:15.633759+0000","last_active":"2024-04-26T20:09:23.353673+0000","last_peered":"2024-04-26T20:09:23.353673+0000","last_clean":"2024-04-26T20:09:23.353673+0000","last_became_active":"2024-04-26T20:08:15.633354+0000","last_became_peered":"2024-04-26T20:08:15.633354+0000","last_unstale":"2024-04-26T20:09:23.353673+0000","last_undegraded":"2024-04-26T20:09:23.353673+0000","last_fullsized":"2024-04-26T20:09:23.353673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:33.394418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.319471+0000","last_change":"2024-04-26T20:09:13.319471+0000","last_active":"2024-04-26T20:09:13.319471+0000","last_peered":"2024-04-26T20:09:13.319471+0000","last_clean":"2024-04-26T20:09:13.319471+0000","last_became_active":"2024-04-26T20:09:13.318292+0000","last_became_peered":"2024-04-26T20:09:13.318292+0000","last_unstale":"2024-04-26T20:09:13.319471+0000","last_undegraded":"2024-04-26T20:09:13.319471+0000","last_fullsized":"2024-04-26T20:09:13.319471+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:41:36.278298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311786+0000","last_change":"2024-04-26T20:08:25.713338+0000","last_active":"2024-04-26T20:09:13.311786+0000","last_peered":"2024-04-26T20:09:13.311786+0000","last_clean":"2024-04-26T20:09:13.311786+0000","last_became_active":"2024-04-26T20:08:25.712905+0000","last_became_peered":"2024-04-26T20:08:25.712905+0000","last_unstale":"2024-04-26T20:09:13.311786+0000","last_undegraded":"2024-04-26T20:09:13.311786+0000","last_fullsized":"2024-04-26T20:09:13.311786+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:54.811957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.363336+0000","last_change":"2024-04-26T20:08:15.633813+0000","last_active":"2024-04-26T20:09:23.363336+0000","last_peered":"2024-04-26T20:09:23.363336+0000","last_clean":"2024-04-26T20:09:23.363336+0000","last_became_active":"2024-04-26T20:08:15.633448+0000","last_became_peered":"2024-04-26T20:08:15.633448+0000","last_unstale":"2024-04-26T20:09:23.363336+0000","last_undegraded":"2024-04-26T20:09:23.363336+0000","last_fullsized":"2024-04-26T20:09:23.363336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:52.909106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313267+0000","last_change":"2024-04-26T20:08:17.659606+0000","last_active":"2024-04-26T20:09:13.313267+0000","last_peered":"2024-04-26T20:09:13.313267+0000","last_clean":"2024-04-26T20:09:13.313267+0000","last_became_active":"2024-04-26T20:08:17.658948+0000","last_became_peered":"2024-04-26T20:08:17.658948+0000","last_unstale":"2024-04-26T20:09:13.313267+0000","last_undegraded":"2024-04-26T20:09:13.313267+0000","last_fullsized":"2024-04-26T20:09:13.313267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:03.616172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269511+0000","last_change":"2024-04-26T20:08:25.700149+0000","last_active":"2024-04-26T20:09:13.269511+0000","last_peered":"2024-04-26T20:09:13.269511+0000","last_clean":"2024-04-26T20:09:13.269511+0000","last_became_active":"2024-04-26T20:08:25.699693+0000","last_became_peered":"2024-04-26T20:08:25.699693+0000","last_unstale":"2024-04-26T20:09:13.269511+0000","last_undegraded":"2024-04-26T20:09:13.269511+0000","last_fullsized":"2024-04-26T20:09:13.269511+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:53.029443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"94'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271791+0000","last_change":"2024-04-26T20:08:15.626386+0000","last_active":"2024-04-26T20:09:13.271791+0000","last_peered":"2024-04-26T20:09:13.271791+0000","last_clean":"2024-04-26T20:09:13.271791+0000","last_became_active":"2024-04-26T20:08:15.626071+0000","last_became_peered":"2024-04-26T20:08:15.626071+0000","last_unstale":"2024-04-26T20:09:13.271791+0000","last_undegraded":"2024-04-26T20:09:13.271791+0000","last_fullsized":"2024-04-26T20:09:13.271791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:49.478038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.276617+0000","last_change":"2024-04-26T20:08:17.668616+0000","last_active":"2024-04-26T20:09:13.276617+0000","last_peered":"2024-04-26T20:09:13.276617+0000","last_clean":"2024-04-26T20:09:13.276617+0000","last_became_active":"2024-04-26T20:08:17.668210+0000","last_became_peered":"2024-04-26T20:08:17.668210+0000","last_unstale":"2024-04-26T20:09:13.276617+0000","last_undegraded":"2024-04-26T20:09:13.276617+0000","last_fullsized":"2024-04-26T20:09:13.276617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:42.219107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313045+0000","last_change":"2024-04-26T20:08:25.699132+0000","last_active":"2024-04-26T20:09:13.313045+0000","last_peered":"2024-04-26T20:09:13.313045+0000","last_clean":"2024-04-26T20:09:13.313045+0000","last_became_active":"2024-04-26T20:08:25.698675+0000","last_became_peered":"2024-04-26T20:08:25.698675+0000","last_unstale":"2024-04-26T20:09:13.313045+0000","last_undegraded":"2024-04-26T20:09:13.313045+0000","last_fullsized":"2024-04-26T20:09:13.313045+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:08:21.950765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'3","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:38.561702+0000","last_change":"2024-04-26T20:08:15.627167+0000","last_active":"2024-04-26T20:09:38.561702+0000","last_peered":"2024-04-26T20:09:38.561702+0000","last_clean":"2024-04-26T20:09:38.561702+0000","last_became_active":"2024-04-26T20:08:15.625928+0000","last_became_peered":"2024-04-26T20:08:15.625928+0000","last_unstale":"2024-04-26T20:09:38.561702+0000","last_undegraded":"2024-04-26T20:09:38.561702+0000","last_fullsized":"2024-04-26T20:09:38.561702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:17:50.361864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313687+0000","last_change":"2024-04-26T20:08:17.652724+0000","last_active":"2024-04-26T20:09:13.313687+0000","last_peered":"2024-04-26T20:09:13.313687+0000","last_clean":"2024-04-26T20:09:13.313687+0000","last_became_active":"2024-04-26T20:08:17.652342+0000","last_became_peered":"2024-04-26T20:08:17.652342+0000","last_unstale":"2024-04-26T20:09:13.313687+0000","last_undegraded":"2024-04-26T20:09:13.313687+0000","last_fullsized":"2024-04-26T20:09:13.313687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:04:37.633017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.274813+0000","last_change":"2024-04-26T20:08:25.703811+0000","last_active":"2024-04-26T20:09:13.274813+0000","last_peered":"2024-04-26T20:09:13.274813+0000","last_clean":"2024-04-26T20:09:13.274813+0000","last_became_active":"2024-04-26T20:08:25.703247+0000","last_became_peered":"2024-04-26T20:08:25.703247+0000","last_unstale":"2024-04-26T20:09:13.274813+0000","last_undegraded":"2024-04-26T20:09:13.274813+0000","last_fullsized":"2024-04-26T20:09:13.274813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:19.605775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.345826+0000","last_change":"2024-04-26T20:08:15.628275+0000","last_active":"2024-04-26T20:09:22.345826+0000","last_peered":"2024-04-26T20:09:22.345826+0000","last_clean":"2024-04-26T20:09:22.345826+0000","last_became_active":"2024-04-26T20:08:15.627115+0000","last_became_peered":"2024-04-26T20:08:15.627115+0000","last_unstale":"2024-04-26T20:09:22.345826+0000","last_undegraded":"2024-04-26T20:09:22.345826+0000","last_fullsized":"2024-04-26T20:09:22.345826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:48.799646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270896+0000","last_change":"2024-04-26T20:08:17.651666+0000","last_active":"2024-04-26T20:09:13.270896+0000","last_peered":"2024-04-26T20:09:13.270896+0000","last_clean":"2024-04-26T20:09:13.270896+0000","last_became_active":"2024-04-26T20:08:17.644762+0000","last_became_peered":"2024-04-26T20:08:17.644762+0000","last_unstale":"2024-04-26T20:09:13.270896+0000","last_undegraded":"2024-04-26T20:09:13.270896+0000","last_fullsized":"2024-04-26T20:09:13.270896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:19.070044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.314326+0000","last_change":"2024-04-26T20:09:13.314326+0000","last_active":"2024-04-26T20:09:13.314326+0000","last_peered":"2024-04-26T20:09:13.314326+0000","last_clean":"2024-04-26T20:09:13.314326+0000","last_became_active":"2024-04-26T20:09:13.313823+0000","last_became_peered":"2024-04-26T20:09:13.313823+0000","last_unstale":"2024-04-26T20:09:13.314326+0000","last_undegraded":"2024-04-26T20:09:13.314326+0000","last_fullsized":"2024-04-26T20:09:13.314326+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:12:55.398462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.346533+0000","last_change":"2024-04-26T20:08:16.123015+0000","last_active":"2024-04-26T20:09:22.346533+0000","last_peered":"2024-04-26T20:09:22.346533+0000","last_clean":"2024-04-26T20:09:22.346533+0000","last_became_active":"2024-04-26T20:08:16.122594+0000","last_became_peered":"2024-04-26T20:08:16.122594+0000","last_unstale":"2024-04-26T20:09:22.346533+0000","last_undegraded":"2024-04-26T20:09:22.346533+0000","last_fullsized":"2024-04-26T20:09:22.346533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:54:11.846029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312071+0000","last_change":"2024-04-26T20:08:17.640882+0000","last_active":"2024-04-26T20:09:13.312071+0000","last_peered":"2024-04-26T20:09:13.312071+0000","last_clean":"2024-04-26T20:09:13.312071+0000","last_became_active":"2024-04-26T20:08:17.640399+0000","last_became_peered":"2024-04-26T20:08:17.640399+0000","last_unstale":"2024-04-26T20:09:13.312071+0000","last_undegraded":"2024-04-26T20:09:13.312071+0000","last_fullsized":"2024-04-26T20:09:13.312071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:05:57.554751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.275493+0000","last_change":"2024-04-26T20:08:25.713003+0000","last_active":"2024-04-26T20:09:13.275493+0000","last_peered":"2024-04-26T20:09:13.275493+0000","last_clean":"2024-04-26T20:09:13.275493+0000","last_became_active":"2024-04-26T20:08:25.711116+0000","last_became_peered":"2024-04-26T20:08:25.711116+0000","last_unstale":"2024-04-26T20:09:13.275493+0000","last_undegraded":"2024-04-26T20:09:13.275493+0000","last_fullsized":"2024-04-26T20:09:13.275493+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:45:50.627388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'30","reported_seq":103,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:39.563408+0000","last_change":"2024-04-26T20:08:15.629344+0000","last_active":"2024-04-26T20:09:39.563408+0000","last_peered":"2024-04-26T20:09:39.563408+0000","last_clean":"2024-04-26T20:09:39.563408+0000","last_became_active":"2024-04-26T20:08:15.628836+0000","last_became_peered":"2024-04-26T20:08:15.628836+0000","last_unstale":"2024-04-26T20:09:39.563408+0000","last_undegraded":"2024-04-26T20:09:39.563408+0000","last_fullsized":"2024-04-26T20:09:39.563408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:05:27.011109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51895,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":70,"num_write":31,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271652+0000","last_change":"2024-04-26T20:08:17.641469+0000","last_active":"2024-04-26T20:09:13.271652+0000","last_peered":"2024-04-26T20:09:13.271652+0000","last_clean":"2024-04-26T20:09:13.271652+0000","last_became_active":"2024-04-26T20:08:17.641147+0000","last_became_peered":"2024-04-26T20:08:17.641147+0000","last_unstale":"2024-04-26T20:09:13.271652+0000","last_undegraded":"2024-04-26T20:09:13.271652+0000","last_fullsized":"2024-04-26T20:09:13.271652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:36:02.885095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272784+0000","last_change":"2024-04-26T20:08:25.708459+0000","last_active":"2024-04-26T20:09:13.272784+0000","last_peered":"2024-04-26T20:09:13.272784+0000","last_clean":"2024-04-26T20:09:13.272784+0000","last_became_active":"2024-04-26T20:08:25.708099+0000","last_became_peered":"2024-04-26T20:08:25.708099+0000","last_unstale":"2024-04-26T20:09:13.272784+0000","last_undegraded":"2024-04-26T20:09:13.272784+0000","last_fullsized":"2024-04-26T20:09:13.272784+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:37.423731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"94'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.359007+0000","last_change":"2024-04-26T20:08:15.627399+0000","last_active":"2024-04-26T20:09:23.359007+0000","last_peered":"2024-04-26T20:09:23.359007+0000","last_clean":"2024-04-26T20:09:23.359007+0000","last_became_active":"2024-04-26T20:08:15.627018+0000","last_became_peered":"2024-04-26T20:08:15.627018+0000","last_unstale":"2024-04-26T20:09:23.359007+0000","last_undegraded":"2024-04-26T20:09:23.359007+0000","last_fullsized":"2024-04-26T20:09:23.359007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:13.491035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279474+0000","last_change":"2024-04-26T20:08:17.669883+0000","last_active":"2024-04-26T20:09:13.279474+0000","last_peered":"2024-04-26T20:09:13.279474+0000","last_clean":"2024-04-26T20:09:13.279474+0000","last_became_active":"2024-04-26T20:08:17.669484+0000","last_became_peered":"2024-04-26T20:08:17.669484+0000","last_unstale":"2024-04-26T20:09:13.279474+0000","last_undegraded":"2024-04-26T20:09:13.279474+0000","last_fullsized":"2024-04-26T20:09:13.279474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:53:28.820771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311593+0000","last_change":"2024-04-26T20:08:25.694352+0000","last_active":"2024-04-26T20:09:13.311593+0000","last_peered":"2024-04-26T20:09:13.311593+0000","last_clean":"2024-04-26T20:09:13.311593+0000","last_became_active":"2024-04-26T20:08:25.694022+0000","last_became_peered":"2024-04-26T20:08:25.694022+0000","last_unstale":"2024-04-26T20:09:13.311593+0000","last_undegraded":"2024-04-26T20:09:13.311593+0000","last_fullsized":"2024-04-26T20:09:13.311593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:23.830964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.345151+0000","last_change":"2024-04-26T20:08:15.625816+0000","last_active":"2024-04-26T20:09:22.345151+0000","last_peered":"2024-04-26T20:09:22.345151+0000","last_clean":"2024-04-26T20:09:22.345151+0000","last_became_active":"2024-04-26T20:08:15.625390+0000","last_became_peered":"2024-04-26T20:08:15.625390+0000","last_unstale":"2024-04-26T20:09:22.345151+0000","last_undegraded":"2024-04-26T20:09:22.345151+0000","last_fullsized":"2024-04-26T20:09:22.345151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:08:57.864795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312947+0000","last_change":"2024-04-26T20:08:17.653016+0000","last_active":"2024-04-26T20:09:13.312947+0000","last_peered":"2024-04-26T20:09:13.312947+0000","last_clean":"2024-04-26T20:09:13.312947+0000","last_became_active":"2024-04-26T20:08:17.652403+0000","last_became_peered":"2024-04-26T20:08:17.652403+0000","last_unstale":"2024-04-26T20:09:13.312947+0000","last_undegraded":"2024-04-26T20:09:13.312947+0000","last_fullsized":"2024-04-26T20:09:13.312947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:02.580052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.280597+0000","last_change":"2024-04-26T20:08:25.717469+0000","last_active":"2024-04-26T20:09:13.280597+0000","last_peered":"2024-04-26T20:09:13.280597+0000","last_clean":"2024-04-26T20:09:13.280597+0000","last_became_active":"2024-04-26T20:08:25.716701+0000","last_became_peered":"2024-04-26T20:08:25.716701+0000","last_unstale":"2024-04-26T20:09:13.280597+0000","last_undegraded":"2024-04-26T20:09:13.280597+0000","last_fullsized":"2024-04-26T20:09:13.280597+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:11:10.991196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'3","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:36.570721+0000","last_change":"2024-04-26T20:08:15.626268+0000","last_active":"2024-04-26T20:09:36.570721+0000","last_peered":"2024-04-26T20:09:36.570721+0000","last_clean":"2024-04-26T20:09:36.570721+0000","last_became_active":"2024-04-26T20:08:15.625839+0000","last_became_peered":"2024-04-26T20:08:15.625839+0000","last_unstale":"2024-04-26T20:09:36.570721+0000","last_undegraded":"2024-04-26T20:09:36.570721+0000","last_fullsized":"2024-04-26T20:09:36.570721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:54.537335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312557+0000","last_change":"2024-04-26T20:08:17.644651+0000","last_active":"2024-04-26T20:09:13.312557+0000","last_peered":"2024-04-26T20:09:13.312557+0000","last_clean":"2024-04-26T20:09:13.312557+0000","last_became_active":"2024-04-26T20:08:17.644357+0000","last_became_peered":"2024-04-26T20:08:17.644357+0000","last_unstale":"2024-04-26T20:09:13.312557+0000","last_undegraded":"2024-04-26T20:09:13.312557+0000","last_fullsized":"2024-04-26T20:09:13.312557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:36.344501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.278589+0000","last_change":"2024-04-26T20:08:25.710016+0000","last_active":"2024-04-26T20:09:13.278589+0000","last_peered":"2024-04-26T20:09:13.278589+0000","last_clean":"2024-04-26T20:09:13.278589+0000","last_became_active":"2024-04-26T20:08:25.709647+0000","last_became_peered":"2024-04-26T20:08:25.709647+0000","last_unstale":"2024-04-26T20:09:13.278589+0000","last_undegraded":"2024-04-26T20:09:13.278589+0000","last_fullsized":"2024-04-26T20:09:13.278589+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:22.434116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"94'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.361458+0000","last_change":"2024-04-26T20:08:16.128347+0000","last_active":"2024-04-26T20:09:23.361458+0000","last_peered":"2024-04-26T20:09:23.361458+0000","last_clean":"2024-04-26T20:09:23.361458+0000","last_became_active":"2024-04-26T20:08:16.127795+0000","last_became_peered":"2024-04-26T20:08:16.127795+0000","last_unstale":"2024-04-26T20:09:23.361458+0000","last_undegraded":"2024-04-26T20:09:23.361458+0000","last_fullsized":"2024-04-26T20:09:23.361458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:21.217202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.314298+0000","last_change":"2024-04-26T20:09:13.314298+0000","last_active":"2024-04-26T20:09:13.314298+0000","last_peered":"2024-04-26T20:09:13.314298+0000","last_clean":"2024-04-26T20:09:13.314298+0000","last_became_active":"2024-04-26T20:09:13.313732+0000","last_became_peered":"2024-04-26T20:09:13.313732+0000","last_unstale":"2024-04-26T20:09:13.314298+0000","last_undegraded":"2024-04-26T20:09:13.314298+0000","last_fullsized":"2024-04-26T20:09:13.314298+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:59:41.782682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272262+0000","last_change":"2024-04-26T20:08:25.700504+0000","last_active":"2024-04-26T20:09:13.272262+0000","last_peered":"2024-04-26T20:09:13.272262+0000","last_clean":"2024-04-26T20:09:13.272262+0000","last_became_active":"2024-04-26T20:08:25.700171+0000","last_became_peered":"2024-04-26T20:08:25.700171+0000","last_unstale":"2024-04-26T20:09:13.272262+0000","last_undegraded":"2024-04-26T20:09:13.272262+0000","last_fullsized":"2024-04-26T20:09:13.272262+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:04.442343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'2","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:24.350667+0000","last_change":"2024-04-26T20:08:15.625738+0000","last_active":"2024-04-26T20:09:24.350667+0000","last_peered":"2024-04-26T20:09:24.350667+0000","last_clean":"2024-04-26T20:09:24.350667+0000","last_became_active":"2024-04-26T20:08:15.625381+0000","last_became_peered":"2024-04-26T20:08:15.625381+0000","last_unstale":"2024-04-26T20:09:24.350667+0000","last_undegraded":"2024-04-26T20:09:24.350667+0000","last_fullsized":"2024-04-26T20:09:24.350667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:52.378477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311750+0000","last_change":"2024-04-26T20:08:17.646682+0000","last_active":"2024-04-26T20:09:13.311750+0000","last_peered":"2024-04-26T20:09:13.311750+0000","last_clean":"2024-04-26T20:09:13.311750+0000","last_became_active":"2024-04-26T20:08:17.646362+0000","last_became_peered":"2024-04-26T20:08:17.646362+0000","last_unstale":"2024-04-26T20:09:13.311750+0000","last_undegraded":"2024-04-26T20:09:13.311750+0000","last_fullsized":"2024-04-26T20:09:13.311750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:52.955883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271299+0000","last_change":"2024-04-26T20:08:25.704580+0000","last_active":"2024-04-26T20:09:13.271299+0000","last_peered":"2024-04-26T20:09:13.271299+0000","last_clean":"2024-04-26T20:09:13.271299+0000","last_became_active":"2024-04-26T20:08:25.704083+0000","last_became_peered":"2024-04-26T20:08:25.704083+0000","last_unstale":"2024-04-26T20:09:13.271299+0000","last_undegraded":"2024-04-26T20:09:13.271299+0000","last_fullsized":"2024-04-26T20:09:13.271299+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:59:33.409501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.275553+0000","last_change":"2024-04-26T20:08:16.131048+0000","last_active":"2024-04-26T20:09:13.275553+0000","last_peered":"2024-04-26T20:09:13.275553+0000","last_clean":"2024-04-26T20:09:13.275553+0000","last_became_active":"2024-04-26T20:08:16.129700+0000","last_became_peered":"2024-04-26T20:08:16.129700+0000","last_unstale":"2024-04-26T20:09:13.275553+0000","last_undegraded":"2024-04-26T20:09:13.275553+0000","last_fullsized":"2024-04-26T20:09:13.275553+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:59:34.814697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.274604+0000","last_change":"2024-04-26T20:08:17.669408+0000","last_active":"2024-04-26T20:09:13.274604+0000","last_peered":"2024-04-26T20:09:13.274604+0000","last_clean":"2024-04-26T20:09:13.274604+0000","last_became_active":"2024-04-26T20:08:17.669102+0000","last_became_peered":"2024-04-26T20:08:17.669102+0000","last_unstale":"2024-04-26T20:09:13.274604+0000","last_undegraded":"2024-04-26T20:09:13.274604+0000","last_fullsized":"2024-04-26T20:09:13.274604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:55:29.812387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279902+0000","last_change":"2024-04-26T20:08:25.692910+0000","last_active":"2024-04-26T20:09:13.279902+0000","last_peered":"2024-04-26T20:09:13.279902+0000","last_clean":"2024-04-26T20:09:13.279902+0000","last_became_active":"2024-04-26T20:08:25.692292+0000","last_became_peered":"2024-04-26T20:08:25.692292+0000","last_unstale":"2024-04-26T20:09:13.279902+0000","last_undegraded":"2024-04-26T20:09:13.279902+0000","last_fullsized":"2024-04-26T20:09:13.279902+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:42:19.483756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313499+0000","last_change":"2024-04-26T20:08:15.633948+0000","last_active":"2024-04-26T20:09:13.313499+0000","last_peered":"2024-04-26T20:09:13.313499+0000","last_clean":"2024-04-26T20:09:13.313499+0000","last_became_active":"2024-04-26T20:08:15.633531+0000","last_became_peered":"2024-04-26T20:08:15.633531+0000","last_unstale":"2024-04-26T20:09:13.313499+0000","last_undegraded":"2024-04-26T20:09:13.313499+0000","last_fullsized":"2024-04-26T20:09:13.313499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:13:15.587856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.274063+0000","last_change":"2024-04-26T20:08:17.663674+0000","last_active":"2024-04-26T20:09:13.274063+0000","last_peered":"2024-04-26T20:09:13.274063+0000","last_clean":"2024-04-26T20:09:13.274063+0000","last_became_active":"2024-04-26T20:08:17.663309+0000","last_became_peered":"2024-04-26T20:08:17.663309+0000","last_unstale":"2024-04-26T20:09:13.274063+0000","last_undegraded":"2024-04-26T20:09:13.274063+0000","last_fullsized":"2024-04-26T20:09:13.274063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:20.661026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279387+0000","last_change":"2024-04-26T20:08:25.698807+0000","last_active":"2024-04-26T20:09:13.279387+0000","last_peered":"2024-04-26T20:09:13.279387+0000","last_clean":"2024-04-26T20:09:13.279387+0000","last_became_active":"2024-04-26T20:08:25.697686+0000","last_became_peered":"2024-04-26T20:08:25.697686+0000","last_unstale":"2024-04-26T20:09:13.279387+0000","last_undegraded":"2024-04-26T20:09:13.279387+0000","last_fullsized":"2024-04-26T20:09:13.279387+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:59:37.099531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272356+0000","last_change":"2024-04-26T20:08:17.645055+0000","last_active":"2024-04-26T20:09:13.272356+0000","last_peered":"2024-04-26T20:09:13.272356+0000","last_clean":"2024-04-26T20:09:13.272356+0000","last_became_active":"2024-04-26T20:08:17.644715+0000","last_became_peered":"2024-04-26T20:08:17.644715+0000","last_unstale":"2024-04-26T20:09:13.272356+0000","last_undegraded":"2024-04-26T20:09:13.272356+0000","last_fullsized":"2024-04-26T20:09:13.272356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:12.508925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'4","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:35.567377+0000","last_change":"2024-04-26T20:08:16.123710+0000","last_active":"2024-04-26T20:09:35.567377+0000","last_peered":"2024-04-26T20:09:35.567377+0000","last_clean":"2024-04-26T20:09:35.567377+0000","last_became_active":"2024-04-26T20:08:16.123046+0000","last_became_peered":"2024-04-26T20:08:16.123046+0000","last_unstale":"2024-04-26T20:09:35.567377+0000","last_undegraded":"2024-04-26T20:09:35.567377+0000","last_fullsized":"2024-04-26T20:09:35.567377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:42.389448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"94'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313027+0000","last_change":"2024-04-26T20:08:25.698716+0000","last_active":"2024-04-26T20:09:13.313027+0000","last_peered":"2024-04-26T20:09:13.313027+0000","last_clean":"2024-04-26T20:09:13.313027+0000","last_became_active":"2024-04-26T20:08:25.698418+0000","last_became_peered":"2024-04-26T20:08:25.698418+0000","last_unstale":"2024-04-26T20:09:13.313027+0000","last_undegraded":"2024-04-26T20:09:13.313027+0000","last_fullsized":"2024-04-26T20:09:13.313027+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:56.290241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.315352+0000","last_change":"2024-04-26T20:08:17.656393+0000","last_active":"2024-04-26T20:09:13.315352+0000","last_peered":"2024-04-26T20:09:13.315352+0000","last_clean":"2024-04-26T20:09:13.315352+0000","last_became_active":"2024-04-26T20:08:17.655914+0000","last_became_peered":"2024-04-26T20:08:17.655914+0000","last_unstale":"2024-04-26T20:09:13.315352+0000","last_undegraded":"2024-04-26T20:09:13.315352+0000","last_fullsized":"2024-04-26T20:09:13.315352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:38:41.459519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"94'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:24.539090+0000","last_change":"2024-04-26T20:08:15.631983+0000","last_active":"2024-04-26T20:09:24.539090+0000","last_peered":"2024-04-26T20:09:24.539090+0000","last_clean":"2024-04-26T20:09:24.539090+0000","last_became_active":"2024-04-26T20:08:15.631625+0000","last_became_peered":"2024-04-26T20:08:15.631625+0000","last_unstale":"2024-04-26T20:09:24.539090+0000","last_undegraded":"2024-04-26T20:09:24.539090+0000","last_fullsized":"2024-04-26T20:09:24.539090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:49.264149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":358,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312356+0000","last_change":"2024-04-26T20:06:47.857115+0000","last_active":"2024-04-26T20:09:13.312356+0000","last_peered":"2024-04-26T20:09:13.312356+0000","last_clean":"2024-04-26T20:09:13.312356+0000","last_became_active":"2024-04-26T20:06:47.856652+0000","last_became_peered":"2024-04-26T20:06:47.856652+0000","last_unstale":"2024-04-26T20:09:13.312356+0000","last_undegraded":"2024-04-26T20:09:13.312356+0000","last_fullsized":"2024-04-26T20:09:13.312356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:02:53.903427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:02:53.903427+0000","last_clean_scrub_stamp":"2024-04-26T20:02:53.903427+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:35:14.359328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.283142+0000","last_change":"2024-04-26T20:09:13.283142+0000","last_active":"2024-04-26T20:09:13.283142+0000","last_peered":"2024-04-26T20:09:13.283142+0000","last_clean":"2024-04-26T20:09:13.283142+0000","last_became_active":"2024-04-26T20:09:13.282806+0000","last_became_peered":"2024-04-26T20:09:13.282806+0000","last_unstale":"2024-04-26T20:09:13.283142+0000","last_undegraded":"2024-04-26T20:09:13.283142+0000","last_fullsized":"2024-04-26T20:09:13.283142+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:39.595299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.278860+0000","last_change":"2024-04-26T20:08:17.654402+0000","last_active":"2024-04-26T20:09:13.278860+0000","last_peered":"2024-04-26T20:09:13.278860+0000","last_clean":"2024-04-26T20:09:13.278860+0000","last_became_active":"2024-04-26T20:08:17.650047+0000","last_became_peered":"2024-04-26T20:08:17.650047+0000","last_unstale":"2024-04-26T20:09:13.278860+0000","last_undegraded":"2024-04-26T20:09:13.278860+0000","last_fullsized":"2024-04-26T20:09:13.278860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:40.225984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.355000+0000","last_change":"2024-04-26T20:08:15.626610+0000","last_active":"2024-04-26T20:09:23.355000+0000","last_peered":"2024-04-26T20:09:23.355000+0000","last_clean":"2024-04-26T20:09:23.355000+0000","last_became_active":"2024-04-26T20:08:15.626162+0000","last_became_peered":"2024-04-26T20:08:15.626162+0000","last_unstale":"2024-04-26T20:09:23.355000+0000","last_undegraded":"2024-04-26T20:09:23.355000+0000","last_fullsized":"2024-04-26T20:09:23.355000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:20.270714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311569+0000","last_change":"2024-04-26T20:08:25.697123+0000","last_active":"2024-04-26T20:09:13.311569+0000","last_peered":"2024-04-26T20:09:13.311569+0000","last_clean":"2024-04-26T20:09:13.311569+0000","last_became_active":"2024-04-26T20:08:25.696773+0000","last_became_peered":"2024-04-26T20:08:25.696773+0000","last_unstale":"2024-04-26T20:09:13.311569+0000","last_undegraded":"2024-04-26T20:09:13.311569+0000","last_fullsized":"2024-04-26T20:09:13.311569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:22:56.642427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"94'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.345678+0000","last_change":"2024-04-26T20:08:16.128206+0000","last_active":"2024-04-26T20:09:22.345678+0000","last_peered":"2024-04-26T20:09:22.345678+0000","last_clean":"2024-04-26T20:09:22.345678+0000","last_became_active":"2024-04-26T20:08:16.127703+0000","last_became_peered":"2024-04-26T20:08:16.127703+0000","last_unstale":"2024-04-26T20:09:22.345678+0000","last_undegraded":"2024-04-26T20:09:22.345678+0000","last_fullsized":"2024-04-26T20:09:22.345678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:39.852292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.280079+0000","last_change":"2024-04-26T20:08:17.652810+0000","last_active":"2024-04-26T20:09:13.280079+0000","last_peered":"2024-04-26T20:09:13.280079+0000","last_clean":"2024-04-26T20:09:13.280079+0000","last_became_active":"2024-04-26T20:08:17.652486+0000","last_became_peered":"2024-04-26T20:08:17.652486+0000","last_unstale":"2024-04-26T20:09:13.280079+0000","last_undegraded":"2024-04-26T20:09:13.280079+0000","last_fullsized":"2024-04-26T20:09:13.280079+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:43.382814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313266+0000","last_change":"2024-04-26T20:08:25.701678+0000","last_active":"2024-04-26T20:09:13.313266+0000","last_peered":"2024-04-26T20:09:13.313266+0000","last_clean":"2024-04-26T20:09:13.313266+0000","last_became_active":"2024-04-26T20:08:25.701283+0000","last_became_peered":"2024-04-26T20:08:25.701283+0000","last_unstale":"2024-04-26T20:09:13.313266+0000","last_undegraded":"2024-04-26T20:09:13.313266+0000","last_fullsized":"2024-04-26T20:09:13.313266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:35:26.774248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:36.675403+0000","last_change":"2024-04-26T20:08:15.634215+0000","last_active":"2024-04-26T20:09:36.675403+0000","last_peered":"2024-04-26T20:09:36.675403+0000","last_clean":"2024-04-26T20:09:36.675403+0000","last_became_active":"2024-04-26T20:08:15.633669+0000","last_became_peered":"2024-04-26T20:08:15.633669+0000","last_unstale":"2024-04-26T20:09:36.675403+0000","last_undegraded":"2024-04-26T20:09:36.675403+0000","last_fullsized":"2024-04-26T20:09:36.675403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:06.178974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.316357+0000","last_change":"2024-04-26T20:08:17.666557+0000","last_active":"2024-04-26T20:09:13.316357+0000","last_peered":"2024-04-26T20:09:13.316357+0000","last_clean":"2024-04-26T20:09:13.316357+0000","last_became_active":"2024-04-26T20:08:17.665552+0000","last_became_peered":"2024-04-26T20:08:17.665552+0000","last_unstale":"2024-04-26T20:09:13.316357+0000","last_undegraded":"2024-04-26T20:09:13.316357+0000","last_fullsized":"2024-04-26T20:09:13.316357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:11.993087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268009+0000","last_change":"2024-04-26T20:08:25.684541+0000","last_active":"2024-04-26T20:09:13.268009+0000","last_peered":"2024-04-26T20:09:13.268009+0000","last_clean":"2024-04-26T20:09:13.268009+0000","last_became_active":"2024-04-26T20:08:25.684159+0000","last_became_peered":"2024-04-26T20:08:25.684159+0000","last_unstale":"2024-04-26T20:09:13.268009+0000","last_undegraded":"2024-04-26T20:09:13.268009+0000","last_fullsized":"2024-04-26T20:09:13.268009+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:31:45.405767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"94'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.343844+0000","last_change":"2024-04-26T20:08:16.002624+0000","last_active":"2024-04-26T20:09:22.343844+0000","last_peered":"2024-04-26T20:09:22.343844+0000","last_clean":"2024-04-26T20:09:22.343844+0000","last_became_active":"2024-04-26T20:08:16.002246+0000","last_became_peered":"2024-04-26T20:08:16.002246+0000","last_unstale":"2024-04-26T20:09:22.343844+0000","last_undegraded":"2024-04-26T20:09:22.343844+0000","last_fullsized":"2024-04-26T20:09:22.343844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:59.393586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.314961+0000","last_change":"2024-04-26T20:09:13.314961+0000","last_active":"2024-04-26T20:09:13.314961+0000","last_peered":"2024-04-26T20:09:13.314961+0000","last_clean":"2024-04-26T20:09:13.314961+0000","last_became_active":"2024-04-26T20:09:13.314533+0000","last_became_peered":"2024-04-26T20:09:13.314533+0000","last_unstale":"2024-04-26T20:09:13.314961+0000","last_undegraded":"2024-04-26T20:09:13.314961+0000","last_fullsized":"2024-04-26T20:09:13.314961+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:29:27.518198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271118+0000","last_change":"2024-04-26T20:08:25.713891+0000","last_active":"2024-04-26T20:09:13.271118+0000","last_peered":"2024-04-26T20:09:13.271118+0000","last_clean":"2024-04-26T20:09:13.271118+0000","last_became_active":"2024-04-26T20:08:25.713480+0000","last_became_peered":"2024-04-26T20:08:25.713480+0000","last_unstale":"2024-04-26T20:09:13.271118+0000","last_undegraded":"2024-04-26T20:09:13.271118+0000","last_fullsized":"2024-04-26T20:09:13.271118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:26.226811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"94'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.346453+0000","last_change":"2024-04-26T20:08:16.123332+0000","last_active":"2024-04-26T20:09:22.346453+0000","last_peered":"2024-04-26T20:09:22.346453+0000","last_clean":"2024-04-26T20:09:22.346453+0000","last_became_active":"2024-04-26T20:08:16.122944+0000","last_became_peered":"2024-04-26T20:08:16.122944+0000","last_unstale":"2024-04-26T20:09:22.346453+0000","last_undegraded":"2024-04-26T20:09:22.346453+0000","last_fullsized":"2024-04-26T20:09:22.346453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:37:23.369867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271089+0000","last_change":"2024-04-26T20:08:17.641693+0000","last_active":"2024-04-26T20:09:13.271089+0000","last_peered":"2024-04-26T20:09:13.271089+0000","last_clean":"2024-04-26T20:09:13.271089+0000","last_became_active":"2024-04-26T20:08:17.641334+0000","last_became_peered":"2024-04-26T20:08:17.641334+0000","last_unstale":"2024-04-26T20:09:13.271089+0000","last_undegraded":"2024-04-26T20:09:13.271089+0000","last_fullsized":"2024-04-26T20:09:13.271089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:05.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.280503+0000","last_change":"2024-04-26T20:08:25.714554+0000","last_active":"2024-04-26T20:09:13.280503+0000","last_peered":"2024-04-26T20:09:13.280503+0000","last_clean":"2024-04-26T20:09:13.280503+0000","last_became_active":"2024-04-26T20:08:25.711975+0000","last_became_peered":"2024-04-26T20:08:25.711975+0000","last_unstale":"2024-04-26T20:09:13.280503+0000","last_undegraded":"2024-04-26T20:09:13.280503+0000","last_fullsized":"2024-04-26T20:09:13.280503+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:20.061288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.346379+0000","last_change":"2024-04-26T20:08:15.634841+0000","last_active":"2024-04-26T20:09:22.346379+0000","last_peered":"2024-04-26T20:09:22.346379+0000","last_clean":"2024-04-26T20:09:22.346379+0000","last_became_active":"2024-04-26T20:08:15.633138+0000","last_became_peered":"2024-04-26T20:08:15.633138+0000","last_unstale":"2024-04-26T20:09:22.346379+0000","last_undegraded":"2024-04-26T20:09:22.346379+0000","last_fullsized":"2024-04-26T20:09:22.346379+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:19.364982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311398+0000","last_change":"2024-04-26T20:08:17.650201+0000","last_active":"2024-04-26T20:09:13.311398+0000","last_peered":"2024-04-26T20:09:13.311398+0000","last_clean":"2024-04-26T20:09:13.311398+0000","last_became_active":"2024-04-26T20:08:17.649819+0000","last_became_peered":"2024-04-26T20:08:17.649819+0000","last_unstale":"2024-04-26T20:09:13.311398+0000","last_undegraded":"2024-04-26T20:09:13.311398+0000","last_fullsized":"2024-04-26T20:09:13.311398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:01.772268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"94'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.316245+0000","last_change":"2024-04-26T20:09:13.315540+0000","last_active":"2024-04-26T20:09:13.316245+0000","last_peered":"2024-04-26T20:09:13.316245+0000","last_clean":"2024-04-26T20:09:13.316245+0000","last_became_active":"2024-04-26T20:09:13.314599+0000","last_became_peered":"2024-04-26T20:09:13.314599+0000","last_unstale":"2024-04-26T20:09:13.316245+0000","last_undegraded":"2024-04-26T20:09:13.316245+0000","last_fullsized":"2024-04-26T20:09:13.316245+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:29.926505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268205+0000","last_change":"2024-04-26T20:08:15.625290+0000","last_active":"2024-04-26T20:09:13.268205+0000","last_peered":"2024-04-26T20:09:13.268205+0000","last_clean":"2024-04-26T20:09:13.268205+0000","last_became_active":"2024-04-26T20:08:15.624310+0000","last_became_peered":"2024-04-26T20:08:15.624310+0000","last_unstale":"2024-04-26T20:09:13.268205+0000","last_undegraded":"2024-04-26T20:09:13.268205+0000","last_fullsized":"2024-04-26T20:09:13.268205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:04:16.736552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.314908+0000","last_change":"2024-04-26T20:08:17.665648+0000","last_active":"2024-04-26T20:09:13.314908+0000","last_peered":"2024-04-26T20:09:13.314908+0000","last_clean":"2024-04-26T20:09:13.314908+0000","last_became_active":"2024-04-26T20:08:17.665205+0000","last_became_peered":"2024-04-26T20:08:17.665205+0000","last_unstale":"2024-04-26T20:09:13.314908+0000","last_undegraded":"2024-04-26T20:09:13.314908+0000","last_fullsized":"2024-04-26T20:09:13.314908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:51.138181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.270979+0000","last_change":"2024-04-26T20:08:25.708815+0000","last_active":"2024-04-26T20:09:13.270979+0000","last_peered":"2024-04-26T20:09:13.270979+0000","last_clean":"2024-04-26T20:09:13.270979+0000","last_became_active":"2024-04-26T20:08:25.708332+0000","last_became_peered":"2024-04-26T20:08:25.708332+0000","last_unstale":"2024-04-26T20:09:13.270979+0000","last_undegraded":"2024-04-26T20:09:13.270979+0000","last_fullsized":"2024-04-26T20:09:13.270979+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:23:19.378956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"94'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:22.344165+0000","last_change":"2024-04-26T20:08:15.626565+0000","last_active":"2024-04-26T20:09:22.344165+0000","last_peered":"2024-04-26T20:09:22.344165+0000","last_clean":"2024-04-26T20:09:22.344165+0000","last_became_active":"2024-04-26T20:08:15.626193+0000","last_became_peered":"2024-04-26T20:08:15.626193+0000","last_unstale":"2024-04-26T20:09:22.344165+0000","last_undegraded":"2024-04-26T20:09:22.344165+0000","last_fullsized":"2024-04-26T20:09:22.344165+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:03:07.382550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313506+0000","last_change":"2024-04-26T20:08:17.658650+0000","last_active":"2024-04-26T20:09:13.313506+0000","last_peered":"2024-04-26T20:09:13.313506+0000","last_clean":"2024-04-26T20:09:13.313506+0000","last_became_active":"2024-04-26T20:08:17.658325+0000","last_became_peered":"2024-04-26T20:08:17.658325+0000","last_unstale":"2024-04-26T20:09:13.313506+0000","last_undegraded":"2024-04-26T20:09:13.313506+0000","last_fullsized":"2024-04-26T20:09:13.313506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:50:27.167289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312351+0000","last_change":"2024-04-26T20:08:25.681040+0000","last_active":"2024-04-26T20:09:13.312351+0000","last_peered":"2024-04-26T20:09:13.312351+0000","last_clean":"2024-04-26T20:09:13.312351+0000","last_became_active":"2024-04-26T20:08:25.680632+0000","last_became_peered":"2024-04-26T20:08:25.680632+0000","last_unstale":"2024-04-26T20:09:13.312351+0000","last_undegraded":"2024-04-26T20:09:13.312351+0000","last_fullsized":"2024-04-26T20:09:13.312351+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:16.017414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:23.361951+0000","last_change":"2024-04-26T20:08:16.123274+0000","last_active":"2024-04-26T20:09:23.361951+0000","last_peered":"2024-04-26T20:09:23.361951+0000","last_clean":"2024-04-26T20:09:23.361951+0000","last_became_active":"2024-04-26T20:08:16.122804+0000","last_became_peered":"2024-04-26T20:08:16.122804+0000","last_unstale":"2024-04-26T20:09:23.361951+0000","last_undegraded":"2024-04-26T20:09:23.361951+0000","last_fullsized":"2024-04-26T20:09:23.361951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:25.235296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313283+0000","last_change":"2024-04-26T20:08:17.641023+0000","last_active":"2024-04-26T20:09:13.313283+0000","last_peered":"2024-04-26T20:09:13.313283+0000","last_clean":"2024-04-26T20:09:13.313283+0000","last_became_active":"2024-04-26T20:08:17.640679+0000","last_became_peered":"2024-04-26T20:08:17.640679+0000","last_unstale":"2024-04-26T20:09:13.313283+0000","last_undegraded":"2024-04-26T20:09:13.313283+0000","last_fullsized":"2024-04-26T20:09:13.313283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:16:37.898958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269339+0000","last_change":"2024-04-26T20:08:25.704251+0000","last_active":"2024-04-26T20:09:13.269339+0000","last_peered":"2024-04-26T20:09:13.269339+0000","last_clean":"2024-04-26T20:09:13.269339+0000","last_became_active":"2024-04-26T20:08:25.702249+0000","last_became_peered":"2024-04-26T20:08:25.702249+0000","last_unstale":"2024-04-26T20:09:13.269339+0000","last_undegraded":"2024-04-26T20:09:13.269339+0000","last_fullsized":"2024-04-26T20:09:13.269339+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:03:12.107793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.272597+0000","last_change":"2024-04-26T20:08:15.636474+0000","last_active":"2024-04-26T20:09:13.272597+0000","last_peered":"2024-04-26T20:09:13.272597+0000","last_clean":"2024-04-26T20:09:13.272597+0000","last_became_active":"2024-04-26T20:08:15.636104+0000","last_became_peered":"2024-04-26T20:08:15.636104+0000","last_unstale":"2024-04-26T20:09:13.272597+0000","last_undegraded":"2024-04-26T20:09:13.272597+0000","last_fullsized":"2024-04-26T20:09:13.272597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:09.514033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312629+0000","last_change":"2024-04-26T20:08:17.665683+0000","last_active":"2024-04-26T20:09:13.312629+0000","last_peered":"2024-04-26T20:09:13.312629+0000","last_clean":"2024-04-26T20:09:13.312629+0000","last_became_active":"2024-04-26T20:08:17.665385+0000","last_became_peered":"2024-04-26T20:08:17.665385+0000","last_unstale":"2024-04-26T20:09:13.312629+0000","last_undegraded":"2024-04-26T20:09:13.312629+0000","last_fullsized":"2024-04-26T20:09:13.312629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:00.826582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271549+0000","last_change":"2024-04-26T20:08:25.683715+0000","last_active":"2024-04-26T20:09:13.271549+0000","last_peered":"2024-04-26T20:09:13.271549+0000","last_clean":"2024-04-26T20:09:13.271549+0000","last_became_active":"2024-04-26T20:08:25.683155+0000","last_became_peered":"2024-04-26T20:08:25.683155+0000","last_unstale":"2024-04-26T20:09:13.271549+0000","last_undegraded":"2024-04-26T20:09:13.271549+0000","last_fullsized":"2024-04-26T20:09:13.271549+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:44.751519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.278720+0000","last_change":"2024-04-26T20:08:15.632519+0000","last_active":"2024-04-26T20:09:13.278720+0000","last_peered":"2024-04-26T20:09:13.278720+0000","last_clean":"2024-04-26T20:09:13.278720+0000","last_became_active":"2024-04-26T20:08:15.631976+0000","last_became_peered":"2024-04-26T20:08:15.631976+0000","last_unstale":"2024-04-26T20:09:13.278720+0000","last_undegraded":"2024-04-26T20:09:13.278720+0000","last_fullsized":"2024-04-26T20:09:13.278720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:39:57.668441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.313301+0000","last_change":"2024-04-26T20:08:17.642480+0000","last_active":"2024-04-26T20:09:13.313301+0000","last_peered":"2024-04-26T20:09:13.313301+0000","last_clean":"2024-04-26T20:09:13.313301+0000","last_became_active":"2024-04-26T20:08:17.642165+0000","last_became_peered":"2024-04-26T20:08:17.642165+0000","last_unstale":"2024-04-26T20:09:13.313301+0000","last_undegraded":"2024-04-26T20:09:13.313301+0000","last_fullsized":"2024-04-26T20:09:13.313301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:05:39.440740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311443+0000","last_change":"2024-04-26T20:08:17.645823+0000","last_active":"2024-04-26T20:09:13.311443+0000","last_peered":"2024-04-26T20:09:13.311443+0000","last_clean":"2024-04-26T20:09:13.311443+0000","last_became_active":"2024-04-26T20:08:17.645413+0000","last_became_peered":"2024-04-26T20:08:17.645413+0000","last_unstale":"2024-04-26T20:09:13.311443+0000","last_undegraded":"2024-04-26T20:09:13.311443+0000","last_fullsized":"2024-04-26T20:09:13.311443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:36:36.622579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268290+0000","last_change":"2024-04-26T20:08:17.641413+0000","last_active":"2024-04-26T20:09:13.268290+0000","last_peered":"2024-04-26T20:09:13.268290+0000","last_clean":"2024-04-26T20:09:13.268290+0000","last_became_active":"2024-04-26T20:08:17.641073+0000","last_became_peered":"2024-04-26T20:08:17.641073+0000","last_unstale":"2024-04-26T20:09:13.268290+0000","last_undegraded":"2024-04-26T20:09:13.268290+0000","last_fullsized":"2024-04-26T20:09:13.268290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:25:03.961399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268049+0000","last_change":"2024-04-26T20:08:17.657336+0000","last_active":"2024-04-26T20:09:13.268049+0000","last_peered":"2024-04-26T20:09:13.268049+0000","last_clean":"2024-04-26T20:09:13.268049+0000","last_became_active":"2024-04-26T20:08:17.656982+0000","last_became_peered":"2024-04-26T20:08:17.656982+0000","last_unstale":"2024-04-26T20:09:13.268049+0000","last_undegraded":"2024-04-26T20:09:13.268049+0000","last_fullsized":"2024-04-26T20:09:13.268049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:10.149705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.269974+0000","last_change":"2024-04-26T20:08:17.662290+0000","last_active":"2024-04-26T20:09:13.269974+0000","last_peered":"2024-04-26T20:09:13.269974+0000","last_clean":"2024-04-26T20:09:13.269974+0000","last_became_active":"2024-04-26T20:08:17.660429+0000","last_became_peered":"2024-04-26T20:08:17.660429+0000","last_unstale":"2024-04-26T20:09:13.269974+0000","last_undegraded":"2024-04-26T20:09:13.269974+0000","last_fullsized":"2024-04-26T20:09:13.269974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:50:03.563066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.284038+0000","last_change":"2024-04-26T20:09:13.284038+0000","last_active":"2024-04-26T20:09:13.284038+0000","last_peered":"2024-04-26T20:09:13.284038+0000","last_clean":"2024-04-26T20:09:13.284038+0000","last_became_active":"2024-04-26T20:09:13.283650+0000","last_became_peered":"2024-04-26T20:09:13.283650+0000","last_unstale":"2024-04-26T20:09:13.284038+0000","last_undegraded":"2024-04-26T20:09:13.284038+0000","last_fullsized":"2024-04-26T20:09:13.284038+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:33.570782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311739+0000","last_change":"2024-04-26T20:08:17.642813+0000","last_active":"2024-04-26T20:09:13.311739+0000","last_peered":"2024-04-26T20:09:13.311739+0000","last_clean":"2024-04-26T20:09:13.311739+0000","last_became_active":"2024-04-26T20:08:17.642018+0000","last_became_peered":"2024-04-26T20:08:17.642018+0000","last_unstale":"2024-04-26T20:09:13.311739+0000","last_undegraded":"2024-04-26T20:09:13.311739+0000","last_fullsized":"2024-04-26T20:09:13.311739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:19.029810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.268589+0000","last_change":"2024-04-26T20:08:17.639030+0000","last_active":"2024-04-26T20:09:13.268589+0000","last_peered":"2024-04-26T20:09:13.268589+0000","last_clean":"2024-04-26T20:09:13.268589+0000","last_became_active":"2024-04-26T20:08:17.638645+0000","last_became_peered":"2024-04-26T20:08:17.638645+0000","last_unstale":"2024-04-26T20:09:13.268589+0000","last_undegraded":"2024-04-26T20:09:13.268589+0000","last_fullsized":"2024-04-26T20:09:13.268589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:58.373648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.316511+0000","last_change":"2024-04-26T20:08:17.666873+0000","last_active":"2024-04-26T20:09:13.316511+0000","last_peered":"2024-04-26T20:09:13.316511+0000","last_clean":"2024-04-26T20:09:13.316511+0000","last_became_active":"2024-04-26T20:08:17.666185+0000","last_became_peered":"2024-04-26T20:08:17.666185+0000","last_unstale":"2024-04-26T20:09:13.316511+0000","last_undegraded":"2024-04-26T20:09:13.316511+0000","last_fullsized":"2024-04-26T20:09:13.316511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:51:17.105637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.271849+0000","last_change":"2024-04-26T20:08:17.666625+0000","last_active":"2024-04-26T20:09:13.271849+0000","last_peered":"2024-04-26T20:09:13.271849+0000","last_clean":"2024-04-26T20:09:13.271849+0000","last_became_active":"2024-04-26T20:08:17.666288+0000","last_became_peered":"2024-04-26T20:08:17.666288+0000","last_unstale":"2024-04-26T20:09:13.271849+0000","last_undegraded":"2024-04-26T20:09:13.271849+0000","last_fullsized":"2024-04-26T20:09:13.271849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:27.051752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.315490+0000","last_change":"2024-04-26T20:08:17.663490+0000","last_active":"2024-04-26T20:09:13.315490+0000","last_peered":"2024-04-26T20:09:13.315490+0000","last_clean":"2024-04-26T20:09:13.315490+0000","last_became_active":"2024-04-26T20:08:17.663131+0000","last_became_peered":"2024-04-26T20:08:17.663131+0000","last_unstale":"2024-04-26T20:09:13.315490+0000","last_undegraded":"2024-04-26T20:09:13.315490+0000","last_fullsized":"2024-04-26T20:09:13.315490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:46.633558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.312640+0000","last_change":"2024-04-26T20:08:17.664749+0000","last_active":"2024-04-26T20:09:13.312640+0000","last_peered":"2024-04-26T20:09:13.312640+0000","last_clean":"2024-04-26T20:09:13.312640+0000","last_became_active":"2024-04-26T20:08:17.664346+0000","last_became_peered":"2024-04-26T20:08:17.664346+0000","last_unstale":"2024-04-26T20:09:13.312640+0000","last_undegraded":"2024-04-26T20:09:13.312640+0000","last_fullsized":"2024-04-26T20:09:13.312640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:27.106743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.279548+0000","last_change":"2024-04-26T20:08:17.662955+0000","last_active":"2024-04-26T20:09:13.279548+0000","last_peered":"2024-04-26T20:09:13.279548+0000","last_clean":"2024-04-26T20:09:13.279548+0000","last_became_active":"2024-04-26T20:08:17.662639+0000","last_became_peered":"2024-04-26T20:08:17.662639+0000","last_unstale":"2024-04-26T20:09:13.279548+0000","last_undegraded":"2024-04-26T20:09:13.279548+0000","last_fullsized":"2024-04-26T20:09:13.279548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:28.346696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T20:09:13.311667+0000","last_change":"2024-04-26T20:08:17.641465+0000","last_active":"2024-04-26T20:09:13.311667+0000","last_peered":"2024-04-26T20:09:13.311667+0000","last_clean":"2024-04-26T20:09:13.311667+0000","last_became_active":"2024-04-26T20:08:17.641145+0000","last_became_peered":"2024-04-26T20:08:17.641145+0000","last_unstale":"2024-04-26T20:09:13.311667+0000","last_undegraded":"2024-04-26T20:09:13.311667+0000","last_fullsized":"2024-04-26T20:09:13.311667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:58.977174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":55896,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":115,"num_read_kb":118,"num_write":160,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":167688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":525894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645348,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28724,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716428,"statfs":{"total":95995035648,"available":95965622272,"internally_reserved":0,"allocated":1748992,"data_stored":1169153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":526130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070512,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":526050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1121959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":521306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":573111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1085440,"data_stored":522694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":574852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":521500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":525956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1683456,"data_stored":1117547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":51895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":53473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T20:09:42.526 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T20:09:42.569 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T20:09:42.584 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T20:09:42.592 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:41 smithi094 ceph-mon[28050]: pgmap v369: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-04-26T20:09:42.599 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:41 smithi094 ceph-mon[28050]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:09:42.605 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:41 smithi103 ceph-mon[32635]: pgmap v369: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2024-04-26T20:09:42.612 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:41 smithi103 ceph-mon[32635]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:09:43.063 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:09:44.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:44 smithi094 ceph-mon[28050]: pgmap v370: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2024-04-26T20:09:45.044 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:44 smithi103 ceph-mon[32635]: pgmap v370: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2024-04-26T20:09:45.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:44 smithi204 ceph-mon[32618]: pgmap v370: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2024-04-26T20:09:45.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.573+0000 7f9615e00700 1 -- 172.21.15.94:0/881370440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 msgr2=0x7f96101027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.573+0000 7f9615e00700 1 --2- 172.21.15.94:0/881370440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 0x7f96101027c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f96000099c0 tx=0x7f9600009cd0 comp rx=0 tx=0).stop 2024-04-26T20:09:45.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.573+0000 7f9615e00700 1 -- 172.21.15.94:0/881370440 shutdown_connections 2024-04-26T20:09:45.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.573+0000 7f9615e00700 1 --2- 172.21.15.94:0/881370440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9610103f40 0x7f961010a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:45.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.573+0000 7f9615e00700 1 --2- 172.21.15.94:0/881370440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 0x7f96101027c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:45.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.573+0000 7f9615e00700 1 --2- 172.21.15.94:0/881370440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96101035a0 0x7f9610103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:45.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.573+0000 7f9615e00700 1 -- 172.21.15.94:0/881370440 >> 172.21.15.94:0/881370440 conn(0x7f96100fd940 msgr2=0x7f96100ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:45.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.573+0000 7f9615e00700 1 -- 172.21.15.94:0/881370440 shutdown_connections 2024-04-26T20:09:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.573+0000 7f9615e00700 1 -- 172.21.15.94:0/881370440 wait complete. 2024-04-26T20:09:45.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.575+0000 7f9615e00700 1 Processor -- start 2024-04-26T20:09:45.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.575+0000 7f9615e00700 1 -- start start 2024-04-26T20:09:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.575+0000 7f9615e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 0x7f9610199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:45.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.575+0000 7f9615e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96101035a0 0x7f9610199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:45.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.575+0000 7f9615e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9610103f40 0x7f96101a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:45.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.575+0000 7f9615e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f961010de10 con 0x7f96101035a0 2024-04-26T20:09:45.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.575+0000 7f9615e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f961010dc90 con 0x7f96101023a0 2024-04-26T20:09:45.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.575+0000 7f9615e00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f961010df90 con 0x7f9610103f40 2024-04-26T20:09:45.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 0x7f9610199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:46.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960e000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96101035a0 0x7f9610199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:46.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 0x7f9610199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:58624/0 (socket says 172.21.15.94:58624) 2024-04-26T20:09:46.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960ea00700 1 -- 172.21.15.94:0/3674137701 learned_addr learned my addr 172.21.15.94:0/3674137701 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:46.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960f400700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9610103f40 0x7f96101a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:46.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960ea00700 1 -- 172.21.15.94:0/3674137701 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9610103f40 msgr2=0x7f96101a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:46.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960ea00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9610103f40 0x7f96101a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:46.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960ea00700 1 -- 172.21.15.94:0/3674137701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96101035a0 msgr2=0x7f9610199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:46.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960ea00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96101035a0 0x7f9610199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:46.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.576+0000 7f960ea00700 1 -- 172.21.15.94:0/3674137701 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96101a1140 con 0x7f96101023a0 2024-04-26T20:09:46.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.577+0000 7f960ea00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 0x7f9610199530 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f96000095c0 tx=0x7f9600003b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:46.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.577+0000 7f9614a00700 1 -- 172.21.15.94:0/3674137701 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600009e60 con 0x7f96101023a0 2024-04-26T20:09:46.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.577+0000 7f9614a00700 1 -- 172.21.15.94:0/3674137701 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9600005570 con 0x7f96101023a0 2024-04-26T20:09:46.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.577+0000 7f9614a00700 1 -- 172.21.15.94:0/3674137701 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960000fc70 con 0x7f96101023a0 2024-04-26T20:09:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.577+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96101a13d0 con 0x7f96101023a0 2024-04-26T20:09:46.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.577+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96101a1860 con 0x7f96101023a0 2024-04-26T20:09:46.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.578+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9610051960 con 0x7f96101023a0 2024-04-26T20:09:46.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.581+0000 7f9614a00700 1 -- 172.21.15.94:0/3674137701 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f960000f460 con 0x7f96101023a0 2024-04-26T20:09:46.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.582+0000 7f9614a00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95f40a3e80 0x7f95f40a6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:46.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.582+0000 7f9614a00700 1 -- 172.21.15.94:0/3674137701 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f96000d1720 con 0x7f96101023a0 2024-04-26T20:09:46.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.582+0000 7f960e000700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95f40a3e80 0x7f95f40a6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.583+0000 7f960e000700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95f40a3e80 0x7f95f40a6340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009c60 tx=0x7f95fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:46.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.585+0000 7f9614a00700 1 -- 172.21.15.94:0/3674137701 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9600013070 con 0x7f96101023a0 2024-04-26T20:09:46.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.914+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f96101a1d30 con 0x7f96101023a0 2024-04-26T20:09:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.924+0000 7f9614a00700 1 -- 172.21.15.94:0/3674137701 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f960009dc70 con 0x7f96101023a0 2024-04-26T20:09:46.456 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-04-26T20:09:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAJCixmo+ahNhAAf+yRS5CZtG0PZER7jJkwYQ== 2024-04-26T20:09:46.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.926+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95f40a3e80 msgr2=0x7f95f40a6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:46.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95f40a3e80 0x7f95f40a6340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009c60 tx=0x7f95fc009260 comp rx=0 tx=0).stop 2024-04-26T20:09:46.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 msgr2=0x7f9610199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:46.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 0x7f9610199530 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f96000095c0 tx=0x7f9600003b70 comp rx=0 tx=0).stop 2024-04-26T20:09:46.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 shutdown_connections 2024-04-26T20:09:46.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9610103f40 0x7f96101a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:46.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f95f40a3e80 0x7f95f40a6340 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:46.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96101023a0 0x7f9610199530 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:46.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 --2- 172.21.15.94:0/3674137701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96101035a0 0x7f9610199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:46.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 >> 172.21.15.94:0/3674137701 conn(0x7f96100fd940 msgr2=0x7f96100ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:46.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 shutdown_connections 2024-04-26T20:09:46.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:45.927+0000 7f9615e00700 1 -- 172.21.15.94:0/3674137701 wait complete. 2024-04-26T20:09:46.900 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:09:46.911 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T20:09:46.917 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T20:09:46.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:46 smithi094 ceph-mon[28050]: pgmap v371: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-04-26T20:09:46.996 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:46 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3674137701' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:09:47.004 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:46 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:09:47.032 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:46 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T20:09:47.043 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:46 smithi103 ceph-mon[32635]: pgmap v371: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-04-26T20:09:47.051 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:46 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3674137701' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:09:47.062 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:46 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:09:47.115 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:46 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T20:09:47.125 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T20:09:47.136 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T20:09:47.185 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T20:09:47.192 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:46 smithi204 ceph-mon[32618]: pgmap v371: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-04-26T20:09:47.198 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:46 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3674137701' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:09:47.204 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:46 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:09:47.211 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:46 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T20:09:47.673 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:09:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:48 smithi094 ceph-mon[28050]: pgmap v372: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T20:09:49.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:48 smithi103 ceph-mon[32635]: pgmap v372: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T20:09:49.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:48 smithi204 ceph-mon[32618]: pgmap v372: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T20:09:49.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.955+0000 7f6edfe00700 1 -- 172.21.15.94:0/3283817438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 msgr2=0x7f6ed81102f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:50.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.955+0000 7f6edfe00700 1 --2- 172.21.15.94:0/3283817438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 0x7f6ed81102f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009f90 tx=0x7f6ed000b330 comp rx=0 tx=0).stop 2024-04-26T20:09:50.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.955+0000 7f6edfe00700 1 -- 172.21.15.94:0/3283817438 shutdown_connections 2024-04-26T20:09:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.955+0000 7f6edfe00700 1 --2- 172.21.15.94:0/3283817438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 0x7f6ed81102f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:50.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.955+0000 7f6edfe00700 1 --2- 172.21.15.94:0/3283817438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ed81094b0 0x7f6ed8109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:50.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.955+0000 7f6edfe00700 1 --2- 172.21.15.94:0/3283817438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101a10 0x7f6ed8101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:50.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.955+0000 7f6edfe00700 1 -- 172.21.15.94:0/3283817438 >> 172.21.15.94:0/3283817438 conn(0x7f6ed8078990 msgr2=0x7f6ed80ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:50.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.955+0000 7f6edfe00700 1 -- 172.21.15.94:0/3283817438 shutdown_connections 2024-04-26T20:09:50.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.955+0000 7f6edfe00700 1 -- 172.21.15.94:0/3283817438 wait complete. 2024-04-26T20:09:50.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.956+0000 7f6edfe00700 1 Processor -- start 2024-04-26T20:09:50.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edfe00700 1 -- start start 2024-04-26T20:09:50.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edfe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101a10 0x7f6ed8102bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:50.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edfe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 0x7f6ed8103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:50.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edfe00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ed81094b0 0x7f6ed8103670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:50.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edfe00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed81135b0 con 0x7f6ed81023b0 2024-04-26T20:09:50.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edfe00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed8113430 con 0x7f6ed8101a10 2024-04-26T20:09:50.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edfe00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed8113730 con 0x7f6ed81094b0 2024-04-26T20:09:50.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edca00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 0x7f6ed8103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:50.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edca00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 0x7f6ed8103130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51716/0 (socket says 172.21.15.94:51716) 2024-04-26T20:09:50.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edca00700 1 -- 172.21.15.94:0/2630272271 learned_addr learned my addr 172.21.15.94:0/2630272271 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:50.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.957+0000 7f6edd400700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101a10 0x7f6ed8102bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:50.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.958+0000 7f6edde00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ed81094b0 0x7f6ed8103670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:50.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.958+0000 7f6edd400700 1 -- 172.21.15.94:0/2630272271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ed81094b0 msgr2=0x7f6ed8103670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:50.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.958+0000 7f6edd400700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ed81094b0 0x7f6ed8103670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:50.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.958+0000 7f6edd400700 1 -- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 msgr2=0x7f6ed8103130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:50.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.958+0000 7f6edd400700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 0x7f6ed8103130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:50.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.958+0000 7f6edd400700 1 -- 172.21.15.94:0/2630272271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed81a7fb0 con 0x7f6ed8101a10 2024-04-26T20:09:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.958+0000 7f6edca00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 0x7f6ed8103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:09:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.958+0000 7f6edd400700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101a10 0x7f6ed8102bf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00e9f0 tx=0x7f6ecc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.959+0000 7f6ed6000700 1 -- 172.21.15.94:0/2630272271 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc00c9c0 con 0x7f6ed8101a10 2024-04-26T20:09:50.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.959+0000 7f6ed6000700 1 -- 172.21.15.94:0/2630272271 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ecc00cb60 con 0x7f6ed8101a10 2024-04-26T20:09:50.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.959+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed81a82a0 con 0x7f6ed8101a10 2024-04-26T20:09:50.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.959+0000 7f6ed6000700 1 -- 172.21.15.94:0/2630272271 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc018680 con 0x7f6ed8101a10 2024-04-26T20:09:50.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.959+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed81a8830 con 0x7f6ed8101a10 2024-04-26T20:09:50.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.960+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed8104d30 con 0x7f6ed8101a10 2024-04-26T20:09:50.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.962+0000 7f6ed6000700 1 -- 172.21.15.94:0/2630272271 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6ecc010430 con 0x7f6ed8101a10 2024-04-26T20:09:51.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.963+0000 7f6ed6000700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6ec00a3d50 0x7f6ec00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.963+0000 7f6ed6000700 1 -- 172.21.15.94:0/2630272271 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f6ecc0d0af0 con 0x7f6ed8101a10 2024-04-26T20:09:51.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.963+0000 7f6edca00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6ec00a3d50 0x7f6ec00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:51.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.964+0000 7f6edca00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6ec00a3d50 0x7f6ec00a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8009750 tx=0x7f6ec8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:51.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:49.966+0000 7f6ed6000700 1 -- 172.21.15.94:0/2630272271 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6ecc09cfb0 con 0x7f6ed8101a10 2024-04-26T20:09:51.163 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:50 smithi204 ceph-mon[32618]: pgmap v373: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T20:09:51.169 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:50 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2630272271' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T20:09:51.177 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:50 smithi103 ceph-mon[32635]: pgmap v373: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T20:09:51.187 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:50 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2630272271' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T20:09:51.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.254+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f6ed802e490 con 0x7f6ed8101a10 2024-04-26T20:09:51.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.259+0000 7f6ed6000700 1 -- 172.21.15.94:0/2630272271 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264997 (secure 0 0 0) 0x7f6ecc09c8b0 con 0x7f6ed8101a10 2024-04-26T20:09:51.219 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:51.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6ec00a3d50 msgr2=0x7f6ec00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:51.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6ec00a3d50 0x7f6ec00a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8009750 tx=0x7f6ec8009310 comp rx=0 tx=0).stop 2024-04-26T20:09:51.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101a10 msgr2=0x7f6ed8102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:51.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101a10 0x7f6ed8102bf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00e9f0 tx=0x7f6ecc00ed00 comp rx=0 tx=0).stop 2024-04-26T20:09:51.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 shutdown_connections 2024-04-26T20:09:51.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6ed81094b0 0x7f6ed8103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6ec00a3d50 0x7f6ec00a6210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:51.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101a10 0x7f6ed8102bf0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:51.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 --2- 172.21.15.94:0/2630272271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ed81023b0 0x7f6ed8103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:51.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 >> 172.21.15.94:0/2630272271 conn(0x7f6ed8078990 msgr2=0x7f6ed8106350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:51.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 shutdown_connections 2024-04-26T20:09:51.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:50.267+0000 7f6edfe00700 1 -- 172.21.15.94:0/2630272271 wait complete. 2024-04-26T20:09:51.577 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:50 smithi094 ceph-mon[28050]: pgmap v373: 129 pgs: 129 active+clean; 631 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T20:09:51.584 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:50 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2630272271' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T20:09:51.591 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":2112565670},{"type":"v1","addr":"172.21.15.94:6801","nonce":2112565670}]},"active_addr":"172.21.15.94:6801/2112565670","active_change":"2024-04-26T20:00:11.413563+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.94:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":251125959}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2678577350}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2837210280}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2490385979}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2711646606}]}]} 2024-04-26T20:09:51.654 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T20:09:51.684 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T20:09:51.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T20:09:52.142 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:09:52.822 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:52 smithi094 ceph-mon[28050]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-26T20:09:53.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:52 smithi103 ceph-mon[32635]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-26T20:09:53.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:52 smithi204 ceph-mon[32618]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-26T20:09:54.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.604+0000 7f8529600700 1 -- 172.21.15.94:0/4166669163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406bad0 msgr2=0x7f85241102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:54.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.604+0000 7f8529600700 1 --2- 172.21.15.94:0/4166669163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406bad0 0x7f85241102e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f851c00b0a0 tx=0x7f851c00b3b0 comp rx=0 tx=0).stop 2024-04-26T20:09:54.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.605+0000 7f8529600700 1 -- 172.21.15.94:0/4166669163 shutdown_connections 2024-04-26T20:09:54.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.605+0000 7f8529600700 1 --2- 172.21.15.94:0/4166669163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406bad0 0x7f85241102e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:54.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.605+0000 7f8529600700 1 --2- 172.21.15.94:0/4166669163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85241094a0 0x7f8524109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:54.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.605+0000 7f8529600700 1 --2- 172.21.15.94:0/4166669163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f852406b130 0x7f852406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:54.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.605+0000 7f8529600700 1 -- 172.21.15.94:0/4166669163 >> 172.21.15.94:0/4166669163 conn(0x7f85240792d0 msgr2=0x7f85240796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:54.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.605+0000 7f8529600700 1 -- 172.21.15.94:0/4166669163 shutdown_connections 2024-04-26T20:09:54.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.605+0000 7f8529600700 1 -- 172.21.15.94:0/4166669163 wait complete. 2024-04-26T20:09:54.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.606+0000 7f8529600700 1 Processor -- start 2024-04-26T20:09:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.606+0000 7f8529600700 1 -- start start 2024-04-26T20:09:54.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8529600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406b130 0x7f852419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:54.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8529600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f852406bad0 0x7f852419c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:54.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8529600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85241094a0 0x7f85241a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:54.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8529600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8524113610 con 0x7f852406b130 2024-04-26T20:09:54.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8529600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8524113490 con 0x7f85241094a0 2024-04-26T20:09:54.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8529600700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8524113790 con 0x7f852406bad0 2024-04-26T20:09:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8522a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406b130 0x7f852419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8522a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406b130 0x7f852419bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36496/0 (socket says 172.21.15.94:36496) 2024-04-26T20:09:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8522a00700 1 -- 172.21.15.94:0/213647073 learned_addr learned my addr 172.21.15.94:0/213647073 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:54.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f851a000700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f852406bad0 0x7f852419c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:54.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8523400700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85241094a0 0x7f85241a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:54.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8522a00700 1 -- 172.21.15.94:0/213647073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f852406bad0 msgr2=0x7f852419c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:54.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8522a00700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f852406bad0 0x7f852419c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8522a00700 1 -- 172.21.15.94:0/213647073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85241094a0 msgr2=0x7f85241a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.607+0000 7f8522a00700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85241094a0 0x7f85241a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:55.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.608+0000 7f8522a00700 1 -- 172.21.15.94:0/213647073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85241a0d50 con 0x7f852406b130 2024-04-26T20:09:55.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.608+0000 7f8522a00700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406b130 0x7f852419bd90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f851400e9c0 tx=0x7f851400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:55.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.608+0000 7f851be00700 1 -- 172.21.15.94:0/213647073 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851400cbd0 con 0x7f852406b130 2024-04-26T20:09:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.608+0000 7f851be00700 1 -- 172.21.15.94:0/213647073 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f851400ee60 con 0x7f852406b130 2024-04-26T20:09:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.608+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85241a1040 con 0x7f852406b130 2024-04-26T20:09:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.608+0000 7f851be00700 1 -- 172.21.15.94:0/213647073 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85140055c0 con 0x7f852406b130 2024-04-26T20:09:55.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.608+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85241a88c0 con 0x7f852406b130 2024-04-26T20:09:55.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.609+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f852406a110 con 0x7f852406b130 2024-04-26T20:09:55.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.613+0000 7f851be00700 1 -- 172.21.15.94:0/213647073 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8514004750 con 0x7f852406b130 2024-04-26T20:09:55.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.614+0000 7f851be00700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85080a3cd0 0x7f85080a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:55.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.615+0000 7f851be00700 1 -- 172.21.15.94:0/213647073 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f85140d0d70 con 0x7f852406b130 2024-04-26T20:09:55.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.615+0000 7f851a000700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85080a3cd0 0x7f85080a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:55.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.616+0000 7f851a000700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85080a3cd0 0x7f85080a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8524079940 tx=0x7f850c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:55.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.617+0000 7f851be00700 1 -- 172.21.15.94:0/213647073 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8514054060 con 0x7f852406b130 2024-04-26T20:09:55.279 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:54 smithi103 ceph-mon[32635]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T20:09:55.347 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:54 smithi204 ceph-mon[32618]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T20:09:55.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.846+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f852419d0a0 con 0x7f852406b130 2024-04-26T20:09:55.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.848+0000 7f851be00700 1 -- 172.21.15.94:0/213647073 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24635 (secure 0 0 0) 0x7f851409d230 con 0x7f852406b130 2024-04-26T20:09:55.385 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:55.390 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:09:13.256835+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:09:55.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.851+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85080a3cd0 msgr2=0x7f85080a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:55.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.851+0000 7f8529600700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85080a3cd0 0x7f85080a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8524079940 tx=0x7f850c009260 comp rx=0 tx=0).stop 2024-04-26T20:09:55.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.851+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406b130 msgr2=0x7f852419bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:55.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.851+0000 7f8529600700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406b130 0x7f852419bd90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f851400e9c0 tx=0x7f851400ecd0 comp rx=0 tx=0).stop 2024-04-26T20:09:55.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.852+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 shutdown_connections 2024-04-26T20:09:55.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.852+0000 7f8529600700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85080a3cd0 0x7f85080a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:55.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.852+0000 7f8529600700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85241094a0 0x7f85241a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:55.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.852+0000 7f8529600700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f852406b130 0x7f852419bd90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:55.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.852+0000 7f8529600700 1 --2- 172.21.15.94:0/213647073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f852406bad0 0x7f852419c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:55.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.852+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 >> 172.21.15.94:0/213647073 conn(0x7f85240792d0 msgr2=0x7f852410e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:55.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.852+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 shutdown_connections 2024-04-26T20:09:55.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:54.852+0000 7f8529600700 1 -- 172.21.15.94:0/213647073 wait complete. 2024-04-26T20:09:55.638 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:54 smithi094 ceph-mon[28050]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T20:09:55.793 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T20:09:55.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T20:09:55.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:55 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/213647073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:56.083 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:55 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/213647073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:56.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:55 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/213647073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:09:56.409 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:09:56.935 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:56 smithi094 ceph-mon[28050]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T20:09:57.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:56 smithi103 ceph-mon[32635]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T20:09:57.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:56 smithi204 ceph-mon[32618]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T20:09:58.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.657+0000 7ff6ff800700 1 -- 172.21.15.94:0/778538061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f810fed0 msgr2=0x7ff6f81122c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:58.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.657+0000 7ff6ff800700 1 --2- 172.21.15.94:0/778538061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f810fed0 0x7ff6f81122c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009990 tx=0x7ff6f0009ca0 comp rx=0 tx=0).stop 2024-04-26T20:09:58.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.658+0000 7ff6ff800700 1 -- 172.21.15.94:0/778538061 shutdown_connections 2024-04-26T20:09:58.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.658+0000 7ff6ff800700 1 --2- 172.21.15.94:0/778538061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f810fed0 0x7ff6f81122c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:58.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.658+0000 7ff6ff800700 1 --2- 172.21.15.94:0/778538061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6f806b130 0x7ff6f806b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:58.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.658+0000 7ff6ff800700 1 --2- 172.21.15.94:0/778538061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f806ba50 0x7ff6f810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:58.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.658+0000 7ff6ff800700 1 -- 172.21.15.94:0/778538061 >> 172.21.15.94:0/778538061 conn(0x7ff6f80792d0 msgr2=0x7ff6f80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:09:58.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.658+0000 7ff6ff800700 1 -- 172.21.15.94:0/778538061 shutdown_connections 2024-04-26T20:09:58.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.658+0000 7ff6ff800700 1 -- 172.21.15.94:0/778538061 wait complete. 2024-04-26T20:09:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.659+0000 7ff6ff800700 1 Processor -- start 2024-04-26T20:09:58.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.659+0000 7ff6ff800700 1 -- start start 2024-04-26T20:09:58.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6ff800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6f806b130 0x7ff6f819bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:59.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6ff800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f806ba50 0x7ff6f819c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:59.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6ff800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f810fed0 0x7ff6f81a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:59.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6ff800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f81157f0 con 0x7ff6f806ba50 2024-04-26T20:09:59.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6ff800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f8115670 con 0x7ff6f810fed0 2024-04-26T20:09:59.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6ff800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f8115970 con 0x7ff6f806b130 2024-04-26T20:09:59.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6f7e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f806ba50 0x7ff6f819c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:59.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6f7e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f806ba50 0x7ff6f819c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36512/0 (socket says 172.21.15.94:36512) 2024-04-26T20:09:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6fd800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f810fed0 0x7ff6f81a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:59.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6f7e00700 1 -- 172.21.15.94:0/1385072888 learned_addr learned my addr 172.21.15.94:0/1385072888 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:09:59.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.660+0000 7ff6fce00700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6f806b130 0x7ff6f819bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:59.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6f7e00700 1 -- 172.21.15.94:0/1385072888 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6f806b130 msgr2=0x7ff6f819bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6f7e00700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6f806b130 0x7ff6f819bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:59.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6f7e00700 1 -- 172.21.15.94:0/1385072888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f810fed0 msgr2=0x7ff6f81a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:59.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6f7e00700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f810fed0 0x7ff6f81a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:59.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6f7e00700 1 -- 172.21.15.94:0/1385072888 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f81a0d20 con 0x7ff6f806ba50 2024-04-26T20:09:59.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6f7e00700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f806ba50 0x7ff6f819c2a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400eb30 tx=0x7ff6e400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:59.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6f5600700 1 -- 172.21.15.94:0/1385072888 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e400cbd0 con 0x7ff6f806ba50 2024-04-26T20:09:59.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6f5600700 1 -- 172.21.15.94:0/1385072888 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6e40044d0 con 0x7ff6f806ba50 2024-04-26T20:09:59.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6f5600700 1 -- 172.21.15.94:0/1385072888 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4003e30 con 0x7ff6f806ba50 2024-04-26T20:09:59.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f81a1010 con 0x7ff6f806ba50 2024-04-26T20:09:59.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.661+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f81a8890 con 0x7ff6f806ba50 2024-04-26T20:09:59.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.662+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f806a110 con 0x7ff6f806ba50 2024-04-26T20:09:59.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.665+0000 7ff6f5600700 1 -- 172.21.15.94:0/1385072888 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff6e4004000 con 0x7ff6f806ba50 2024-04-26T20:09:59.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.666+0000 7ff6f5600700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff6e00a3d50 0x7ff6e00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:09:59.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.666+0000 7ff6f5600700 1 -- 172.21.15.94:0/1385072888 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7ff6e40d0dd0 con 0x7ff6f806ba50 2024-04-26T20:09:59.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.666+0000 7ff6fce00700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff6e00a3d50 0x7ff6e00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:09:59.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.667+0000 7ff6fce00700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff6e00a3d50 0x7ff6e00a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00f540 tx=0x7ff6ec00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:09:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.669+0000 7ff6f5600700 1 -- 172.21.15.94:0/1385072888 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff6e4054060 con 0x7ff6f806ba50 2024-04-26T20:09:59.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.899+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6f802e490 con 0x7ff6f806ba50 2024-04-26T20:09:59.684 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:58 smithi094 ceph-mon[28050]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T20:09:59.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.902+0000 7ff6f5600700 1 -- 172.21.15.94:0/1385072888 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24635 (secure 0 0 0) 0x7ff6e40042f0 con 0x7ff6f806ba50 2024-04-26T20:09:59.721 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:09:59.731 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:09:13.256835+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:09:59.744 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:58 smithi204 ceph-mon[32618]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T20:09:59.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:58 smithi103 ceph-mon[32635]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T20:09:59.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff6e00a3d50 msgr2=0x7ff6e00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:59.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff6e00a3d50 0x7ff6e00a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00f540 tx=0x7ff6ec00b040 comp rx=0 tx=0).stop 2024-04-26T20:09:59.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f806ba50 msgr2=0x7ff6f819c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:09:59.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f806ba50 0x7ff6f819c2a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400eb30 tx=0x7ff6e400ee40 comp rx=0 tx=0).stop 2024-04-26T20:09:59.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 shutdown_connections 2024-04-26T20:09:59.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff6e00a3d50 0x7ff6e00a6210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:09:59.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f810fed0 0x7ff6f81a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:00.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6f806b130 0x7ff6f819bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:00.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 --2- 172.21.15.94:0/1385072888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff6f806ba50 0x7ff6f819c2a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:00.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 >> 172.21.15.94:0/1385072888 conn(0x7ff6f80792d0 msgr2=0x7ff6f81113c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 shutdown_connections 2024-04-26T20:10:00.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:09:58.905+0000 7ff6ff800700 1 -- 172.21.15.94:0/1385072888 wait complete. 2024-04-26T20:10:00.173 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T20:10:00.186 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T20:10:00.201 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T20:10:00.215 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T20:10:00.228 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T20:10:00.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T20:10:00.256 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T20:10:00.268 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T20:10:00.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T20:10:00.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T20:10:00.407 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T20:10:00.470 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T20:10:00.480 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:09:59 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1385072888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:10:00.490 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:09:59 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1385072888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:10:00.498 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:09:59 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1385072888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:10:00.990 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:00 smithi094 ceph-mon[28050]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T20:10:01.007 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:00 smithi094 ceph-mon[28050]: overall HEALTH_OK 2024-04-26T20:10:01.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:00 smithi103 ceph-mon[32635]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T20:10:01.042 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:00 smithi103 ceph-mon[32635]: overall HEALTH_OK 2024-04-26T20:10:01.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:00 smithi204 ceph-mon[32618]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-26T20:10:01.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:00 smithi204 ceph-mon[32618]: overall HEALTH_OK 2024-04-26T20:10:01.495 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:01.520 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:01.631 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:01.738 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:01.772 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:02.497 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:02.731 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:02 smithi094 ceph-mon[28050]: pgmap v379: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T20:10:03.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:02 smithi103 ceph-mon[32635]: pgmap v379: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T20:10:03.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:02 smithi204 ceph-mon[32618]: pgmap v379: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-26T20:10:04.886 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:04.920 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:04.985 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:04 smithi094 ceph-mon[28050]: pgmap v380: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:10:05.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:04 smithi103 ceph-mon[32635]: pgmap v380: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:10:05.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:04 smithi204 ceph-mon[32618]: pgmap v380: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:10:05.141 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:06.176 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:06.277 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:06.291 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:06.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:06 smithi094 ceph-mon[28050]: pgmap v381: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:10:07.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:06 smithi103 ceph-mon[32635]: pgmap v381: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:10:07.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:06 smithi204 ceph-mon[32618]: pgmap v381: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:10:08.910 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:08 smithi094 ceph-mon[28050]: pgmap v382: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T20:10:09.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:08 smithi103 ceph-mon[32635]: pgmap v382: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T20:10:09.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:08 smithi204 ceph-mon[32618]: pgmap v382: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T20:10:10.822 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:10 smithi094 ceph-mon[28050]: pgmap v383: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T20:10:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:10 smithi103 ceph-mon[32635]: pgmap v383: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T20:10:11.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:10 smithi204 ceph-mon[32618]: pgmap v383: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T20:10:12.027 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:11 smithi094 ceph-mon[28050]: pgmap v384: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:10:12.105 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:11 smithi103 ceph-mon[32635]: pgmap v384: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:10:12.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:11 smithi204 ceph-mon[32618]: pgmap v384: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:10:12.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.381+0000 7f1c39a00700 1 -- 172.21.15.94:0/493507717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c3410fed0 msgr2=0x7f1c341122c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:12.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.381+0000 7f1c39a00700 1 --2- 172.21.15.94:0/493507717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c3410fed0 0x7f1c341122c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c009990 tx=0x7f1c2c009ca0 comp rx=0 tx=0).stop 2024-04-26T20:10:12.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.381+0000 7f1c39a00700 1 -- 172.21.15.94:0/493507717 shutdown_connections 2024-04-26T20:10:12.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.381+0000 7f1c39a00700 1 --2- 172.21.15.94:0/493507717 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c3410fed0 0x7f1c341122c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:12.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.381+0000 7f1c39a00700 1 --2- 172.21.15.94:0/493507717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c3406b130 0x7f1c3406b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:12.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.381+0000 7f1c39a00700 1 --2- 172.21.15.94:0/493507717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c3406ba50 0x7f1c3410f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:12.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.381+0000 7f1c39a00700 1 -- 172.21.15.94:0/493507717 >> 172.21.15.94:0/493507717 conn(0x7f1c340792d0 msgr2=0x7f1c340796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:12.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.381+0000 7f1c39a00700 1 -- 172.21.15.94:0/493507717 shutdown_connections 2024-04-26T20:10:12.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.382+0000 7f1c39a00700 1 -- 172.21.15.94:0/493507717 wait complete. 2024-04-26T20:10:12.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.383+0000 7f1c39a00700 1 Processor -- start 2024-04-26T20:10:12.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.383+0000 7f1c39a00700 1 -- start start 2024-04-26T20:10:12.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c39a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c3406b130 0x7f1c341a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:12.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c39a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c3406ba50 0x7f1c341a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:12.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c39a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c3410fed0 0x7f1c3419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:12.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c39a00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34115830 con 0x7f1c3406b130 2024-04-26T20:10:12.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c39a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c341156b0 con 0x7f1c3410fed0 2024-04-26T20:10:12.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c39a00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1c341159b0 con 0x7f1c3406ba50 2024-04-26T20:10:12.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c32a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c3406b130 0x7f1c341a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:12.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c32a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c3406b130 0x7f1c341a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54926/0 (socket says 172.21.15.94:54926) 2024-04-26T20:10:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c32a00700 1 -- 172.21.15.94:0/1044810083 learned_addr learned my addr 172.21.15.94:0/1044810083 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:12.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c33400700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c3410fed0 0x7f1c3419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:12.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c32000700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c3406ba50 0x7f1c341a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:12.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c32a00700 1 -- 172.21.15.94:0/1044810083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c3406ba50 msgr2=0x7f1c341a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:12.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.384+0000 7f1c32a00700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c3406ba50 0x7f1c341a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c32a00700 1 -- 172.21.15.94:0/1044810083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c3410fed0 msgr2=0x7f1c3419bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:12.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c32a00700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c3410fed0 0x7f1c3419bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:12.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c32a00700 1 -- 172.21.15.94:0/1044810083 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3419c410 con 0x7f1c3406b130 2024-04-26T20:10:12.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c32000700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c3406ba50 0x7f1c341a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:10:12.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c32a00700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c3406b130 0x7f1c341a1a60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400ef10 tx=0x7f1c2400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:12.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c2b400700 1 -- 172.21.15.94:0/1044810083 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400c9c0 con 0x7f1c3406b130 2024-04-26T20:10:12.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c2b400700 1 -- 172.21.15.94:0/1044810083 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c24019070 con 0x7f1c3406b130 2024-04-26T20:10:12.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c2b400700 1 -- 172.21.15.94:0/1044810083 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24014480 con 0x7f1c3406b130 2024-04-26T20:10:13.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c3419c6a0 con 0x7f1c3406b130 2024-04-26T20:10:13.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.385+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c341a8810 con 0x7f1c3406b130 2024-04-26T20:10:13.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.386+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1c34063b70 con 0x7f1c3406b130 2024-04-26T20:10:13.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.389+0000 7f1c2b400700 1 -- 172.21.15.94:0/1044810083 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1c24012050 con 0x7f1c3406b130 2024-04-26T20:10:13.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.390+0000 7f1c2b400700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1c180a3cd0 0x7f1c180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:13.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.390+0000 7f1c2b400700 1 -- 172.21.15.94:0/1044810083 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f1c240d13a0 con 0x7f1c3406b130 2024-04-26T20:10:13.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.390+0000 7f1c32000700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1c180a3cd0 0x7f1c180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:13.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.390+0000 7f1c2b400700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] conn(0x7f1c180aee30 0x7f1c180b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:13.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.390+0000 7f1c2b400700 1 -- 172.21.15.94:0/1044810083 --> [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c24022e50 con 0x7f1c180aee30 2024-04-26T20:10:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.390+0000 7f1c2b400700 1 -- 172.21.15.94:0/1044810083 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f1c240d1750 con 0x7f1c3406b130 2024-04-26T20:10:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.391+0000 7f1c33400700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] conn(0x7f1c180aee30 0x7f1c180b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:13.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.391+0000 7f1c32000700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1c180a3cd0 0x7f1c180a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c3419d280 tx=0x7f1c20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:13.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.391+0000 7f1c33400700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] conn(0x7f1c180aee30 0x7f1c180b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:13.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.392+0000 7f1c2b400700 1 -- 172.21.15.94:0/1044810083 <== osd.11 v2:172.21.15.204:6824/1860743299 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c24022e50 con 0x7f1c180aee30 2024-04-26T20:10:13.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.410+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 --> [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1c34063ff0 con 0x7f1c180aee30 2024-04-26T20:10:13.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.411+0000 7f1c2b400700 1 -- 172.21.15.94:0/1044810083 <== osd.11 v2:172.21.15.204:6824/1860743299 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1c34063ff0 con 0x7f1c180aee30 2024-04-26T20:10:13.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.411+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 >> [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] conn(0x7f1c180aee30 msgr2=0x7f1c180b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:13.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.411+0000 7f1c39a00700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] conn(0x7f1c180aee30 0x7f1c180b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:13.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.412+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1c180a3cd0 msgr2=0x7f1c180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:13.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.412+0000 7f1c39a00700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1c180a3cd0 0x7f1c180a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c3419d280 tx=0x7f1c20009260 comp rx=0 tx=0).stop 2024-04-26T20:10:13.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.412+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c3406b130 msgr2=0x7f1c341a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:13.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.412+0000 7f1c39a00700 1 --2- 172.21.15.94:0/1044810083 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c3406b130 0x7f1c341a1a60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400ef10 tx=0x7f1c2400c3a0 comp rx=0 tx=0).stop 2024-04-26T20:10:13.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.412+0000 7f1c33400700 1 -- 172.21.15.94:0/1044810083 reap_dead start 2024-04-26T20:10:13.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.412+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 shutdown_connections 2024-04-26T20:10:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.412+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 >> 172.21.15.94:0/1044810083 conn(0x7f1c340792d0 msgr2=0x7f1c34111380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:13.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.412+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 shutdown_connections 2024-04-26T20:10:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:12.412+0000 7f1c39a00700 1 -- 172.21.15.94:0/1044810083 wait complete. 2024-04-26T20:10:13.388 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-26T20:10:13.399 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-26T20:10:13.408 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-26T20:10:13.417 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-04-26T20:10:13.481 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:12 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:10:13.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-26T20:10:13.562 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-26T20:10:13.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-26T20:10:13.689 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-04-26T20:10:13.700 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:12 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:10:13.708 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-26T20:10:13.715 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-26T20:10:13.721 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-26T20:10:13.728 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-04-26T20:10:13.736 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:12 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:10:13.876 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-26T20:10:13.920 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-26T20:10:13.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-26T20:10:13.935 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-04-26T20:10:13.942 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:13 smithi094 ceph-mon[28050]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T20:10:13.949 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:13 smithi094 ceph-mon[28050]: pgmap v386: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T20:10:13.956 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:14.009 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:14.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-26T20:10:14.088 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-26T20:10:14.123 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-26T20:10:14.131 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-04-26T20:10:14.140 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:13 smithi103 ceph-mon[32635]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T20:10:14.152 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:13 smithi103 ceph-mon[32635]: pgmap v386: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T20:10:14.170 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:14.179 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:14.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-26T20:10:14.241 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-26T20:10:14.250 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-26T20:10:14.259 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-04-26T20:10:14.269 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:13 smithi204 ceph-mon[32618]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T20:10:14.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:13 smithi204 ceph-mon[32618]: pgmap v386: 129 pgs: 129 active+clean; 636 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T20:10:14.290 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:14.339 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.026 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:14 smithi094 ceph-mon[28050]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T20:10:15.041 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.100 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.109 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.117 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.125 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:14 smithi103 ceph-mon[32635]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T20:10:15.131 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.142 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.148 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.154 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.161 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:14 smithi204 ceph-mon[32618]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T20:10:15.168 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.174 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:15.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:15 smithi094 ceph-mon[28050]: pgmap v388: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T20:10:16.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:15 smithi103 ceph-mon[32635]: pgmap v388: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T20:10:16.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:15 smithi204 ceph-mon[32618]: pgmap v388: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T20:10:17.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.360+0000 7f38c8000700 1 -- 172.21.15.94:0/202857078 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c0075650 msgr2=0x7f38c0075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:17.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.360+0000 7f38c8000700 1 --2- 172.21.15.94:0/202857078 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c0075650 0x7f38c0075a30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f38b4009990 tx=0x7f38b4009ca0 comp rx=0 tx=0).stop 2024-04-26T20:10:17.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.360+0000 7f38c8000700 1 -- 172.21.15.94:0/202857078 shutdown_connections 2024-04-26T20:10:17.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.360+0000 7f38c8000700 1 --2- 172.21.15.94:0/202857078 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c010fe20 0x7f38c0112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:17.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.360+0000 7f38c8000700 1 --2- 172.21.15.94:0/202857078 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c0075650 0x7f38c0075a30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.360+0000 7f38c8000700 1 --2- 172.21.15.94:0/202857078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c0075f70 0x7f38c010f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:17.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.360+0000 7f38c8000700 1 -- 172.21.15.94:0/202857078 >> 172.21.15.94:0/202857078 conn(0x7f38c007a580 msgr2=0x7f38c007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:17.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.361+0000 7f38c8000700 1 -- 172.21.15.94:0/202857078 shutdown_connections 2024-04-26T20:10:17.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.361+0000 7f38c8000700 1 -- 172.21.15.94:0/202857078 wait complete. 2024-04-26T20:10:17.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.362+0000 7f38c8000700 1 Processor -- start 2024-04-26T20:10:17.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.362+0000 7f38c8000700 1 -- start start 2024-04-26T20:10:17.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.362+0000 7f38c8000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0075650 0x7f38c019bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:17.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.362+0000 7f38c8000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c0075f70 0x7f38c019c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:17.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.362+0000 7f38c8000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c010fe20 0x7f38c01a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c8000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f38c01157f0 con 0x7f38c010fe20 2024-04-26T20:10:17.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c8000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38c0115670 con 0x7f38c0075f70 2024-04-26T20:10:17.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c8000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f38c0115970 con 0x7f38c0075650 2024-04-26T20:10:17.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c6000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c010fe20 0x7f38c01a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:17.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c6000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c010fe20 0x7f38c01a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54972/0 (socket says 172.21.15.94:54972) 2024-04-26T20:10:17.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c6000700 1 -- 172.21.15.94:0/56108738 learned_addr learned my addr 172.21.15.94:0/56108738 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:17.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c5600700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0075650 0x7f38c019bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:17.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c4c00700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c0075f70 0x7f38c019c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:17.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c6000700 1 -- 172.21.15.94:0/56108738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0075650 msgr2=0x7f38c019bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:17.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c6000700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38c0075650 0x7f38c019bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:17.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c6000700 1 -- 172.21.15.94:0/56108738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c0075f70 msgr2=0x7f38c019c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:17.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c6000700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c0075f70 0x7f38c019c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.363+0000 7f38c6000700 1 -- 172.21.15.94:0/56108738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c01a0d20 con 0x7f38c010fe20 2024-04-26T20:10:17.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.364+0000 7f38c6000700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c010fe20 0x7f38c01a05e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38b800ef90 tx=0x7f38b800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:17.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.364+0000 7f38be000700 1 -- 172.21.15.94:0/56108738 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b80092e0 con 0x7f38c010fe20 2024-04-26T20:10:17.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.364+0000 7f38be000700 1 -- 172.21.15.94:0/56108738 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38b8004bb0 con 0x7f38c010fe20 2024-04-26T20:10:17.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.364+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c01a0fb0 con 0x7f38c010fe20 2024-04-26T20:10:17.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.364+0000 7f38be000700 1 -- 172.21.15.94:0/56108738 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8004280 con 0x7f38c010fe20 2024-04-26T20:10:17.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.364+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c01a8890 con 0x7f38c010fe20 2024-04-26T20:10:17.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.365+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f38c0063bb0 con 0x7f38c010fe20 2024-04-26T20:10:17.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.368+0000 7f38be000700 1 -- 172.21.15.94:0/56108738 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f38b801b030 con 0x7f38c010fe20 2024-04-26T20:10:18.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.369+0000 7f38be000700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f38a80a3cd0 0x7f38a80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:18.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.369+0000 7f38be000700 1 -- 172.21.15.94:0/56108738 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f38b80d0b80 con 0x7f38c010fe20 2024-04-26T20:10:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.369+0000 7f38c5600700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f38a80a3cd0 0x7f38a80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.369+0000 7f38be000700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] conn(0x7f38a80af030 0x7f38a80b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:18.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.369+0000 7f38be000700 1 -- 172.21.15.94:0/56108738 --> [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38b801ee50 con 0x7f38a80af030 2024-04-26T20:10:18.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.369+0000 7f38be000700 1 -- 172.21.15.94:0/56108738 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f38b80d0f30 con 0x7f38c010fe20 2024-04-26T20:10:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.370+0000 7f38c4c00700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] conn(0x7f38a80af030 0x7f38a80b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.370+0000 7f38c5600700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f38a80a3cd0 0x7f38a80a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f38b40098e0 tx=0x7f38b4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:18.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.370+0000 7f38c4c00700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] conn(0x7f38a80af030 0x7f38a80b14d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.371+0000 7f38be000700 1 -- 172.21.15.94:0/56108738 <== osd.7 v2:172.21.15.103:6816/1379914194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38b801ee50 con 0x7f38a80af030 2024-04-26T20:10:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.389+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 --> [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f38c0064030 con 0x7f38a80af030 2024-04-26T20:10:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.390+0000 7f38be000700 1 -- 172.21.15.94:0/56108738 <== osd.7 v2:172.21.15.103:6816/1379914194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f38c0064030 con 0x7f38a80af030 2024-04-26T20:10:18.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 >> [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] conn(0x7f38a80af030 msgr2=0x7f38a80b14d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:18.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] conn(0x7f38a80af030 0x7f38a80b14d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:18.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f38a80a3cd0 msgr2=0x7f38a80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:18.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f38a80a3cd0 0x7f38a80a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f38b40098e0 tx=0x7f38b4005960 comp rx=0 tx=0).stop 2024-04-26T20:10:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c010fe20 msgr2=0x7f38c01a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:18.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 --2- 172.21.15.94:0/56108738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c010fe20 0x7f38c01a05e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38b800ef90 tx=0x7f38b800c3a0 comp rx=0 tx=0).stop 2024-04-26T20:10:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c6000700 1 -- 172.21.15.94:0/56108738 reap_dead start 2024-04-26T20:10:18.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 shutdown_connections 2024-04-26T20:10:18.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 >> 172.21.15.94:0/56108738 conn(0x7f38c007a580 msgr2=0x7f38c0101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:18.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 shutdown_connections 2024-04-26T20:10:18.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.391+0000 7f38c8000700 1 -- 172.21.15.94:0/56108738 wait complete. 2024-04-26T20:10:18.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.491+0000 7f584e400700 1 -- 172.21.15.94:0/818840469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f584810fe20 msgr2=0x7f5848112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:18.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.491+0000 7f584e400700 1 --2- 172.21.15.94:0/818840469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f584810fe20 0x7f5848112210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5840009f90 tx=0x7f584000b330 comp rx=0 tx=0).stop 2024-04-26T20:10:18.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.491+0000 7f584e400700 1 -- 172.21.15.94:0/818840469 shutdown_connections 2024-04-26T20:10:18.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.491+0000 7f584e400700 1 --2- 172.21.15.94:0/818840469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f584810fe20 0x7f5848112210 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:18.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.491+0000 7f584e400700 1 --2- 172.21.15.94:0/818840469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5848075650 0x7f5848075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:18.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.491+0000 7f584e400700 1 --2- 172.21.15.94:0/818840469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5848075f70 0x7f584810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:18.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.491+0000 7f584e400700 1 -- 172.21.15.94:0/818840469 >> 172.21.15.94:0/818840469 conn(0x7f584807a580 msgr2=0x7f584807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:18.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.491+0000 7f584e400700 1 -- 172.21.15.94:0/818840469 shutdown_connections 2024-04-26T20:10:18.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.492+0000 7f584e400700 1 -- 172.21.15.94:0/818840469 wait complete. 2024-04-26T20:10:18.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.493+0000 7f584e400700 1 Processor -- start 2024-04-26T20:10:18.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.493+0000 7f584e400700 1 -- start start 2024-04-26T20:10:18.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.493+0000 7f584e400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5848075650 0x7f58481a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:18.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.493+0000 7f584e400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5848075f70 0x7f58481a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:18.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.493+0000 7f584e400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f584810fe20 0x7f584819baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:18.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.493+0000 7f584e400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5848115800 con 0x7f584810fe20 2024-04-26T20:10:18.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.493+0000 7f584e400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5848115680 con 0x7f5848075f70 2024-04-26T20:10:18.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.493+0000 7f584e400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5848115980 con 0x7f5848075650 2024-04-26T20:10:18.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.493+0000 7f5846a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5848075f70 0x7f58481a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:18.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5846a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5848075f70 0x7f58481a1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:46426/0 (socket says 172.21.15.94:46426) 2024-04-26T20:10:18.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5846a00700 1 -- 172.21.15.94:0/1346580722 learned_addr learned my addr 172.21.15.94:0/1346580722 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:18.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5847e00700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f584810fe20 0x7f584819baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:18.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5847400700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5848075650 0x7f58481a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:18.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5846a00700 1 -- 172.21.15.94:0/1346580722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5848075650 msgr2=0x7f58481a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:18.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5846a00700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5848075650 0x7f58481a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:18.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5846a00700 1 -- 172.21.15.94:0/1346580722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f584810fe20 msgr2=0x7f584819baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:18.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5846a00700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f584810fe20 0x7f584819baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:18.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5846a00700 1 -- 172.21.15.94:0/1346580722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584819c200 con 0x7f5848075f70 2024-04-26T20:10:18.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5847e00700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f584810fe20 0x7f584819baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:10:18.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.494+0000 7f5846a00700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5848075f70 0x7f58481a1e20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f583400ebf0 tx=0x7f583400ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:18.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.495+0000 7f583fe00700 1 -- 172.21.15.94:0/1346580722 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583400cd40 con 0x7f5848075f70 2024-04-26T20:10:18.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.495+0000 7f583fe00700 1 -- 172.21.15.94:0/1346580722 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58340108e0 con 0x7f5848075f70 2024-04-26T20:10:18.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.495+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584819c4f0 con 0x7f5848075f70 2024-04-26T20:10:18.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.495+0000 7f583fe00700 1 -- 172.21.15.94:0/1346580722 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5834018840 con 0x7f5848075f70 2024-04-26T20:10:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.495+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58481a87b0 con 0x7f5848075f70 2024-04-26T20:10:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.495+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5848063bb0 con 0x7f5848075f70 2024-04-26T20:10:19.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.499+0000 7f583fe00700 1 -- 172.21.15.94:0/1346580722 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f583401f020 con 0x7f5848075f70 2024-04-26T20:10:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.500+0000 7f583fe00700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f582c0a3d50 0x7f582c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:19.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.500+0000 7f583fe00700 1 -- 172.21.15.94:0/1346580722 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f58340d12e0 con 0x7f5848075f70 2024-04-26T20:10:19.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.500+0000 7f5847400700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f582c0a3d50 0x7f582c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.501+0000 7f583fe00700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] conn(0x7f582c0af0b0 0x7f582c0b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:19.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.501+0000 7f583fe00700 1 -- 172.21.15.94:0/1346580722 --> [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f582c0b1c20 con 0x7f582c0af0b0 2024-04-26T20:10:19.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.501+0000 7f583fe00700 1 -- 172.21.15.94:0/1346580722 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f58340d1730 con 0x7f5848075f70 2024-04-26T20:10:19.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.501+0000 7f5847400700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f582c0a3d50 0x7f582c0a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f584819cbe0 tx=0x7f5838009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:19.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.501+0000 7f5847e00700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] conn(0x7f582c0af0b0 0x7f582c0b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.502+0000 7f5847e00700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] conn(0x7f582c0af0b0 0x7f582c0b1550 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:19.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.503+0000 7f583fe00700 1 -- 172.21.15.94:0/1346580722 <== osd.1 v2:172.21.15.103:6800/1497163956 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f582c0b1c20 con 0x7f582c0af0b0 2024-04-26T20:10:19.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.521+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 --> [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5848063fb0 con 0x7f582c0af0b0 2024-04-26T20:10:19.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.521+0000 7f583fe00700 1 -- 172.21.15.94:0/1346580722 <== osd.1 v2:172.21.15.103:6800/1497163956 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5848063fb0 con 0x7f582c0af0b0 2024-04-26T20:10:19.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 >> [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] conn(0x7f582c0af0b0 msgr2=0x7f582c0b1550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] conn(0x7f582c0af0b0 0x7f582c0b1550 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f582c0a3d50 msgr2=0x7f582c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f582c0a3d50 0x7f582c0a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f584819cbe0 tx=0x7f5838009260 comp rx=0 tx=0).stop 2024-04-26T20:10:19.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5848075f70 msgr2=0x7f58481a1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:19.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 --2- 172.21.15.94:0/1346580722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5848075f70 0x7f58481a1e20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f583400ebf0 tx=0x7f583400ef00 comp rx=0 tx=0).stop 2024-04-26T20:10:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f5847e00700 1 -- 172.21.15.94:0/1346580722 reap_dead start 2024-04-26T20:10:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 shutdown_connections 2024-04-26T20:10:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 >> 172.21.15.94:0/1346580722 conn(0x7f584807a580 msgr2=0x7f58481005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:19.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 shutdown_connections 2024-04-26T20:10:19.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:17.522+0000 7f584e400700 1 -- 172.21.15.94:0/1346580722 wait complete. 2024-04-26T20:10:19.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.848+0000 7fb883000700 1 -- 172.21.15.94:0/3445189480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c108670 msgr2=0x7fb87c110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:19.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.848+0000 7fb883000700 1 --2- 172.21.15.94:0/3445189480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c108670 0x7fb87c110240 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb87400b0d0 tx=0x7fb87400b3e0 comp rx=0 tx=0).stop 2024-04-26T20:10:19.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.848+0000 7fb883000700 1 -- 172.21.15.94:0/3445189480 shutdown_connections 2024-04-26T20:10:19.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.848+0000 7fb883000700 1 --2- 172.21.15.94:0/3445189480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c108670 0x7fb87c110240 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:19.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.848+0000 7fb883000700 1 --2- 172.21.15.94:0/3445189480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb87c103590 0x7fb87c103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:19.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.848+0000 7fb883000700 1 --2- 172.21.15.94:0/3445189480 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb87c103f40 0x7fb87c107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:19.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.848+0000 7fb883000700 1 -- 172.21.15.94:0/3445189480 >> 172.21.15.94:0/3445189480 conn(0x7fb87c0fee40 msgr2=0x7fb87c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:19.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.848+0000 7fb883000700 1 -- 172.21.15.94:0/3445189480 shutdown_connections 2024-04-26T20:10:19.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.848+0000 7fb883000700 1 -- 172.21.15.94:0/3445189480 wait complete. 2024-04-26T20:10:19.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.849+0000 7fb883000700 1 Processor -- start 2024-04-26T20:10:19.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb883000700 1 -- start start 2024-04-26T20:10:19.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb883000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb87c103590 0x7fb87c1a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:19.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb883000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c103f40 0x7fb87c1a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:19.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb883000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb87c108670 0x7fb87c19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:19.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb883000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c113560 con 0x7fb87c103f40 2024-04-26T20:10:19.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb883000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c1133e0 con 0x7fb87c103590 2024-04-26T20:10:19.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb883000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c1136e0 con 0x7fb87c108670 2024-04-26T20:10:19.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb881200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c103f40 0x7fb87c1a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:19.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb881c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb87c103590 0x7fb87c1a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:19.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb881c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb87c103590 0x7fb87c1a18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:46378/0 (socket says 172.21.15.94:46378) 2024-04-26T20:10:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.850+0000 7fb881c00700 1 -- 172.21.15.94:0/1651633388 learned_addr learned my addr 172.21.15.94:0/1651633388 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:19.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.851+0000 7fb882600700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb87c108670 0x7fb87c19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:19.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.851+0000 7fb881c00700 1 -- 172.21.15.94:0/1651633388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb87c108670 msgr2=0x7fb87c19ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:19.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.851+0000 7fb881c00700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb87c108670 0x7fb87c19ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:19.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.851+0000 7fb881c00700 1 -- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c103f40 msgr2=0x7fb87c1a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:19.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.851+0000 7fb881c00700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c103f40 0x7fb87c1a1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:19.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.851+0000 7fb881c00700 1 -- 172.21.15.94:0/1651633388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c19c1e0 con 0x7fb87c103590 2024-04-26T20:10:19.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.851+0000 7fb881c00700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb87c103590 0x7fb87c1a18c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb87000e9a0 tx=0x7fb87000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:19.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.851+0000 7fb87a000700 1 -- 172.21.15.94:0/1651633388 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87000ca10 con 0x7fb87c103590 2024-04-26T20:10:19.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.852+0000 7fb87a000700 1 -- 172.21.15.94:0/1651633388 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb87000cbb0 con 0x7fb87c103590 2024-04-26T20:10:19.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.852+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87c19c4d0 con 0x7fb87c103590 2024-04-26T20:10:19.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.852+0000 7fb87a000700 1 -- 172.21.15.94:0/1651633388 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870018680 con 0x7fb87c103590 2024-04-26T20:10:19.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.852+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87c1a8790 con 0x7fb87c103590 2024-04-26T20:10:19.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.852+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb87c076930 con 0x7fb87c103590 2024-04-26T20:10:20.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.855+0000 7fb87a000700 1 -- 172.21.15.94:0/1651633388 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb870010430 con 0x7fb87c103590 2024-04-26T20:10:20.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.856+0000 7fb87a000700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8640a3d50 0x7fb8640a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.856+0000 7fb87a000700 1 -- 172.21.15.94:0/1651633388 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fb8700d0ff0 con 0x7fb87c103590 2024-04-26T20:10:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.856+0000 7fb881200700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8640a3d50 0x7fb8640a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:20.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.856+0000 7fb87a000700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] conn(0x7fb8640af0b0 0x7fb8640b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:20.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.856+0000 7fb87a000700 1 -- 172.21.15.94:0/1651633388 --> [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb870019ec0 con 0x7fb8640af0b0 2024-04-26T20:10:20.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.856+0000 7fb87a000700 1 -- 172.21.15.94:0/1651633388 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fb8700d1420 con 0x7fb87c103590 2024-04-26T20:10:20.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.857+0000 7fb882600700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] conn(0x7fb8640af0b0 0x7fb8640b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:20.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.857+0000 7fb881200700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8640a3d50 0x7fb8640a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb87c101810 tx=0x7fb868009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:20.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.857+0000 7fb882600700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] conn(0x7fb8640af0b0 0x7fb8640b1550 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:20.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.858+0000 7fb87a000700 1 -- 172.21.15.94:0/1651633388 <== osd.9 v2:172.21.15.94:6826/4013917902 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb870019ec0 con 0x7fb8640af0b0 2024-04-26T20:10:20.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.876+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 --> [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb87c104a70 con 0x7fb8640af0b0 2024-04-26T20:10:20.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.876+0000 7fb87a000700 1 -- 172.21.15.94:0/1651633388 <== osd.9 v2:172.21.15.94:6826/4013917902 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb87c104a70 con 0x7fb8640af0b0 2024-04-26T20:10:20.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.876+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] conn(0x7fb8640af0b0 msgr2=0x7fb8640b1550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:20.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.876+0000 7fb883000700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] conn(0x7fb8640af0b0 0x7fb8640b1550 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:20.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.877+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8640a3d50 msgr2=0x7fb8640a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:20.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.877+0000 7fb883000700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb8640a3d50 0x7fb8640a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb87c101810 tx=0x7fb868009260 comp rx=0 tx=0).stop 2024-04-26T20:10:20.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.877+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb87c103590 msgr2=0x7fb87c1a18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:20.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.877+0000 7fb883000700 1 --2- 172.21.15.94:0/1651633388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb87c103590 0x7fb87c1a18c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb87000e9a0 tx=0x7fb87000ecb0 comp rx=0 tx=0).stop 2024-04-26T20:10:20.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.877+0000 7fb882600700 1 -- 172.21.15.94:0/1651633388 reap_dead start 2024-04-26T20:10:20.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.877+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 shutdown_connections 2024-04-26T20:10:20.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.877+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 >> 172.21.15.94:0/1651633388 conn(0x7fb87c0fee40 msgr2=0x7fb87c10e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:20.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.877+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 shutdown_connections 2024-04-26T20:10:20.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:16.877+0000 7fb883000700 1 -- 172.21.15.94:0/1651633388 wait complete. 2024-04-26T20:10:20.683 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:18 smithi094 ceph-mon[28050]: pgmap v389: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2024-04-26T20:10:20.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.825+0000 7f809a200700 1 -- 172.21.15.94:0/3170599017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8094109480 msgr2=0x7f8094109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:20.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.825+0000 7f809a200700 1 --2- 172.21.15.94:0/3170599017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8094109480 0x7f8094109860 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8084009990 tx=0x7f8084009ca0 comp rx=0 tx=0).stop 2024-04-26T20:10:20.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.825+0000 7f809a200700 1 -- 172.21.15.94:0/3170599017 shutdown_connections 2024-04-26T20:10:20.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.825+0000 7f809a200700 1 --2- 172.21.15.94:0/3170599017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8094103ea0 0x7f80941102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:20.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.825+0000 7f809a200700 1 --2- 172.21.15.94:0/3170599017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8094109480 0x7f8094109860 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:20.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.825+0000 7f809a200700 1 --2- 172.21.15.94:0/3170599017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80941034e0 0x7f8094103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:20.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.825+0000 7f809a200700 1 -- 172.21.15.94:0/3170599017 >> 172.21.15.94:0/3170599017 conn(0x7f809407a5e0 msgr2=0x7f809407a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:20.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.825+0000 7f809a200700 1 -- 172.21.15.94:0/3170599017 shutdown_connections 2024-04-26T20:10:20.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.825+0000 7f809a200700 1 -- 172.21.15.94:0/3170599017 wait complete. 2024-04-26T20:10:20.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.827+0000 7f809a200700 1 Processor -- start 2024-04-26T20:10:20.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.827+0000 7f809a200700 1 -- start start 2024-04-26T20:10:20.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.827+0000 7f809a200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80941034e0 0x7f80941a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:20.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.827+0000 7f809a200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8094103ea0 0x7f80941a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:20.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.827+0000 7f809a200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8094109480 0x7f809419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:20.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.827+0000 7f809a200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f80941135b0 con 0x7f8094103ea0 2024-04-26T20:10:20.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.827+0000 7f809a200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8094113430 con 0x7f80941034e0 2024-04-26T20:10:21.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.827+0000 7f809a200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8094113730 con 0x7f8094109480 2024-04-26T20:10:21.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.827+0000 7f8093400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80941034e0 0x7f80941a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:21.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8093400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80941034e0 0x7f80941a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:46360/0 (socket says 172.21.15.94:46360) 2024-04-26T20:10:21.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8093400700 1 -- 172.21.15.94:0/2749487723 learned_addr learned my addr 172.21.15.94:0/2749487723 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:21.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8093e00700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8094109480 0x7f809419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:21.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8092a00700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8094103ea0 0x7f80941a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:21.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8093400700 1 -- 172.21.15.94:0/2749487723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8094109480 msgr2=0x7f809419baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:21.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8093400700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8094109480 0x7f809419baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8093400700 1 -- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8094103ea0 msgr2=0x7f80941a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:21.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8093400700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8094103ea0 0x7f80941a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:21.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8093400700 1 -- 172.21.15.94:0/2749487723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809419c250 con 0x7f80941034e0 2024-04-26T20:10:21.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.828+0000 7f8093400700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80941034e0 0x7f80941a1930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8084000c00 tx=0x7f808400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:21.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.829+0000 7f808be00700 1 -- 172.21.15.94:0/2749487723 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8084027070 con 0x7f80941034e0 2024-04-26T20:10:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.829+0000 7f808be00700 1 -- 172.21.15.94:0/2749487723 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8084022470 con 0x7f80941034e0 2024-04-26T20:10:21.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.829+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809419c4e0 con 0x7f80941034e0 2024-04-26T20:10:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.829+0000 7f808be00700 1 -- 172.21.15.94:0/2749487723 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8084017650 con 0x7f80941034e0 2024-04-26T20:10:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.829+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80941a8800 con 0x7f80941034e0 2024-04-26T20:10:21.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.830+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8094101d70 con 0x7f80941034e0 2024-04-26T20:10:21.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.832+0000 7f808be00700 1 -- 172.21.15.94:0/2749487723 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f808402a070 con 0x7f80941034e0 2024-04-26T20:10:21.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.833+0000 7f808be00700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f80780a3d50 0x7f80780a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:21.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.833+0000 7f808be00700 1 -- 172.21.15.94:0/2749487723 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f80840d1a50 con 0x7f80941034e0 2024-04-26T20:10:21.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.834+0000 7f8092a00700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f80780a3d50 0x7f80780a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:21.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.834+0000 7f808be00700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] conn(0x7f80780af0b0 0x7f80780b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:21.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.834+0000 7f808be00700 1 -- 172.21.15.94:0/2749487723 --> [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80840095d0 con 0x7f80780af0b0 2024-04-26T20:10:21.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.834+0000 7f808be00700 1 -- 172.21.15.94:0/2749487723 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f80840d1e00 con 0x7f80941034e0 2024-04-26T20:10:21.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.834+0000 7f8093e00700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] conn(0x7f80780af0b0 0x7f80780b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:21.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.834+0000 7f8092a00700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f80780a3d50 0x7f80780a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8080009680 tx=0x7f8080009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.835+0000 7f8093e00700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] conn(0x7f80780af0b0 0x7f80780b1550 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:21.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.836+0000 7f808be00700 1 -- 172.21.15.94:0/2749487723 <== osd.0 v2:172.21.15.94:6802/3381923901 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80840095d0 con 0x7f80780af0b0 2024-04-26T20:10:21.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.853+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 --> [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8094104b30 con 0x7f80780af0b0 2024-04-26T20:10:21.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.854+0000 7f808be00700 1 -- 172.21.15.94:0/2749487723 <== osd.0 v2:172.21.15.94:6802/3381923901 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8094104b30 con 0x7f80780af0b0 2024-04-26T20:10:21.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.854+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] conn(0x7f80780af0b0 msgr2=0x7f80780b1550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.854+0000 7f809a200700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] conn(0x7f80780af0b0 0x7f80780b1550 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:21.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.855+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f80780a3d50 msgr2=0x7f80780a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:21.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.855+0000 7f809a200700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f80780a3d50 0x7f80780a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8080009680 tx=0x7f8080009260 comp rx=0 tx=0).stop 2024-04-26T20:10:21.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.855+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80941034e0 msgr2=0x7f80941a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:21.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.855+0000 7f809a200700 1 --2- 172.21.15.94:0/2749487723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80941034e0 0x7f80941a1930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8084000c00 tx=0x7f808400bfd0 comp rx=0 tx=0).stop 2024-04-26T20:10:21.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.855+0000 7f8093e00700 1 -- 172.21.15.94:0/2749487723 reap_dead start 2024-04-26T20:10:21.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.855+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 shutdown_connections 2024-04-26T20:10:21.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.855+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 >> 172.21.15.94:0/2749487723 conn(0x7f809407a5e0 msgr2=0x7f8094101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:21.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.855+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 shutdown_connections 2024-04-26T20:10:21.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:14.855+0000 7f809a200700 1 -- 172.21.15.94:0/2749487723 wait complete. 2024-04-26T20:10:21.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.584+0000 7f6a65c00700 1 -- 172.21.15.94:0/4142398628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a60109a50 msgr2=0x7f6a60109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.584+0000 7f6a65c00700 1 --2- 172.21.15.94:0/4142398628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a60109a50 0x7f6a60109ed0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009990 tx=0x7f6a48009ca0 comp rx=0 tx=0).stop 2024-04-26T20:10:21.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.584+0000 7f6a65c00700 1 -- 172.21.15.94:0/4142398628 shutdown_connections 2024-04-26T20:10:21.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.584+0000 7f6a65c00700 1 --2- 172.21.15.94:0/4142398628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a6010a410 0x7f6a60116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.584+0000 7f6a65c00700 1 --2- 172.21.15.94:0/4142398628 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6010fa50 0x7f6a6010fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:22.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.584+0000 7f6a65c00700 1 --2- 172.21.15.94:0/4142398628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a60109a50 0x7f6a60109ed0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:22.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.584+0000 7f6a65c00700 1 -- 172.21.15.94:0/4142398628 >> 172.21.15.94:0/4142398628 conn(0x7f6a60076bf0 msgr2=0x7f6a60108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:22.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.585+0000 7f6a65c00700 1 -- 172.21.15.94:0/4142398628 shutdown_connections 2024-04-26T20:10:22.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.585+0000 7f6a65c00700 1 -- 172.21.15.94:0/4142398628 wait complete. 2024-04-26T20:10:22.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.585+0000 7f6a65c00700 1 Processor -- start 2024-04-26T20:10:22.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a65c00700 1 -- start start 2024-04-26T20:10:22.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a65c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a60109a50 0x7f6a601a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:22.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a65c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6010a410 0x7f6a601a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:22.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a65c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a6010fa50 0x7f6a601a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:22.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a65c00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60119c50 con 0x7f6a60109a50 2024-04-26T20:10:22.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a65c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60119ad0 con 0x7f6a6010fa50 2024-04-26T20:10:22.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a65c00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6a60119dd0 con 0x7f6a6010a410 2024-04-26T20:10:22.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a5ea00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a60109a50 0x7f6a601a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:22.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a5ea00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a60109a50 0x7f6a601a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55022/0 (socket says 172.21.15.94:55022) 2024-04-26T20:10:22.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.586+0000 7f6a5ea00700 1 -- 172.21.15.94:0/1016823674 learned_addr learned my addr 172.21.15.94:0/1016823674 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.587+0000 7f6a5f400700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a6010fa50 0x7f6a601a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:22.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.587+0000 7f6a57e00700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6010a410 0x7f6a601a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:22.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.587+0000 7f6a5ea00700 1 -- 172.21.15.94:0/1016823674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6010a410 msgr2=0x7f6a601a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:22.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.587+0000 7f6a5ea00700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a6010a410 0x7f6a601a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:22.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.587+0000 7f6a5ea00700 1 -- 172.21.15.94:0/1016823674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a6010fa50 msgr2=0x7f6a601a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:22.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.587+0000 7f6a5ea00700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a6010fa50 0x7f6a601a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:22.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.587+0000 7f6a5ea00700 1 -- 172.21.15.94:0/1016823674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a601a5170 con 0x7f6a60109a50 2024-04-26T20:10:22.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.587+0000 7f6a5ea00700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a60109a50 0x7f6a601a0160 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6a5000e9c0 tx=0x7f6a5000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:22.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.587+0000 7f6a57400700 1 -- 172.21.15.94:0/1016823674 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5000cbd0 con 0x7f6a60109a50 2024-04-26T20:10:22.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.588+0000 7f6a57400700 1 -- 172.21.15.94:0/1016823674 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a5000ee60 con 0x7f6a60109a50 2024-04-26T20:10:22.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.588+0000 7f6a57400700 1 -- 172.21.15.94:0/1016823674 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a500055c0 con 0x7f6a60109a50 2024-04-26T20:10:22.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.588+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a601a5460 con 0x7f6a60109a50 2024-04-26T20:10:22.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.588+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a601acce0 con 0x7f6a60109a50 2024-04-26T20:10:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.589+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6a60063af0 con 0x7f6a60109a50 2024-04-26T20:10:22.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.591+0000 7f6a57400700 1 -- 172.21.15.94:0/1016823674 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6a50004750 con 0x7f6a60109a50 2024-04-26T20:10:22.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.592+0000 7f6a57400700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6a440a3cd0 0x7f6a440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:22.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.592+0000 7f6a57400700 1 -- 172.21.15.94:0/1016823674 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f6a500d0fc0 con 0x7f6a60109a50 2024-04-26T20:10:22.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.592+0000 7f6a57e00700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6a440a3cd0 0x7f6a440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:22.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.593+0000 7f6a57400700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] conn(0x7f6a440af030 0x7f6a440b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:22.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.593+0000 7f6a57400700 1 -- 172.21.15.94:0/1016823674 --> [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a5001ee50 con 0x7f6a440af030 2024-04-26T20:10:22.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.593+0000 7f6a57400700 1 -- 172.21.15.94:0/1016823674 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f6a500d1370 con 0x7f6a60109a50 2024-04-26T20:10:22.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.593+0000 7f6a5f400700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] conn(0x7f6a440af030 0x7f6a440b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:22.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.593+0000 7f6a57e00700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6a440a3cd0 0x7f6a440a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a480098e0 tx=0x7f6a48005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:22.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.594+0000 7f6a5f400700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] conn(0x7f6a440af030 0x7f6a440b14d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:22.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.594+0000 7f6a57400700 1 -- 172.21.15.94:0/1016823674 <== osd.2 v2:172.21.15.204:6800/1401040530 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a5001ee50 con 0x7f6a440af030 2024-04-26T20:10:22.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.612+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 --> [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6a60063ef0 con 0x7f6a440af030 2024-04-26T20:10:22.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.613+0000 7f6a57400700 1 -- 172.21.15.94:0/1016823674 <== osd.2 v2:172.21.15.204:6800/1401040530 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6a60063ef0 con 0x7f6a440af030 2024-04-26T20:10:22.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 >> [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] conn(0x7f6a440af030 msgr2=0x7f6a440b14d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:22.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] conn(0x7f6a440af030 0x7f6a440b14d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6a440a3cd0 msgr2=0x7f6a440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:22.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6a440a3cd0 0x7f6a440a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a480098e0 tx=0x7f6a48005960 comp rx=0 tx=0).stop 2024-04-26T20:10:22.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a60109a50 msgr2=0x7f6a601a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:22.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 --2- 172.21.15.94:0/1016823674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a60109a50 0x7f6a601a0160 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6a5000e9c0 tx=0x7f6a5000ecd0 comp rx=0 tx=0).stop 2024-04-26T20:10:22.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a5f400700 1 -- 172.21.15.94:0/1016823674 reap_dead start 2024-04-26T20:10:22.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 shutdown_connections 2024-04-26T20:10:22.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 >> 172.21.15.94:0/1016823674 conn(0x7f6a60076bf0 msgr2=0x7f6a60077210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:22.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 shutdown_connections 2024-04-26T20:10:22.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:18.614+0000 7f6a65c00700 1 -- 172.21.15.94:0/1016823674 wait complete. 2024-04-26T20:10:22.889 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:20 smithi094 ceph-mon[28050]: pgmap v390: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 199 B/s, 1 objects/s recovering 2024-04-26T20:10:22.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:18 smithi204 ceph-mon[32618]: pgmap v389: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2024-04-26T20:10:22.911 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:20 smithi204 ceph-mon[32618]: pgmap v390: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 199 B/s, 1 objects/s recovering 2024-04-26T20:10:22.921 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:18 smithi103 ceph-mon[32635]: pgmap v389: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2024-04-26T20:10:22.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:20 smithi103 ceph-mon[32635]: pgmap v390: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 199 B/s, 1 objects/s recovering 2024-04-26T20:10:22.943 INFO:teuthology.orchestra.run.smithi094.stdout:77309411419 2024-04-26T20:10:22.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T20:10:22.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.580+0000 7ff74c600700 1 -- 172.21.15.94:0/1459539809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff744105640 msgr2=0x7ff744105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.580+0000 7ff74c600700 1 --2- 172.21.15.94:0/1459539809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff744105640 0x7ff744105ac0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff7300098f0 tx=0x7ff730009c00 comp rx=0 tx=0).stop 2024-04-26T20:10:22.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.580+0000 7ff74c600700 1 -- 172.21.15.94:0/1459539809 shutdown_connections 2024-04-26T20:10:22.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.580+0000 7ff74c600700 1 --2- 172.21.15.94:0/1459539809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff744106000 0x7ff744112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:22.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.580+0000 7ff74c600700 1 --2- 172.21.15.94:0/1459539809 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff74410b640 0x7ff74410ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:23.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.580+0000 7ff74c600700 1 --2- 172.21.15.94:0/1459539809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff744105640 0x7ff744105ac0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:23.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.580+0000 7ff74c600700 1 -- 172.21.15.94:0/1459539809 >> 172.21.15.94:0/1459539809 conn(0x7ff744101130 msgr2=0x7ff744103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:23.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.580+0000 7ff74c600700 1 -- 172.21.15.94:0/1459539809 shutdown_connections 2024-04-26T20:10:23.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.580+0000 7ff74c600700 1 -- 172.21.15.94:0/1459539809 wait complete. 2024-04-26T20:10:23.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.581+0000 7ff74c600700 1 Processor -- start 2024-04-26T20:10:23.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.581+0000 7ff74c600700 1 -- start start 2024-04-26T20:10:23.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff74c600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff744105640 0x7ff74419bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:23.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff74c600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff744106000 0x7ff74419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:23.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff74c600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff74410b640 0x7ff7441a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:23.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff74c600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff7441158d0 con 0x7ff744106000 2024-04-26T20:10:23.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff74c600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff744115750 con 0x7ff74410b640 2024-04-26T20:10:23.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff74c600700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff744115a50 con 0x7ff744105640 2024-04-26T20:10:23.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff749200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff744106000 0x7ff74419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:23.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff749c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff744105640 0x7ff74419bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:23.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff749200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff744106000 0x7ff74419c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42788/0 (socket says 172.21.15.94:42788) 2024-04-26T20:10:23.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff749200700 1 -- 172.21.15.94:0/2415299807 learned_addr learned my addr 172.21.15.94:0/2415299807 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:23.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.582+0000 7ff74a600700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff74410b640 0x7ff7441a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:23.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff749200700 1 -- 172.21.15.94:0/2415299807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff744105640 msgr2=0x7ff74419bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:23.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff749200700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff744105640 0x7ff74419bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:23.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff749200700 1 -- 172.21.15.94:0/2415299807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff74410b640 msgr2=0x7ff7441a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:23.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff749200700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff74410b640 0x7ff7441a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:23.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff749200700 1 -- 172.21.15.94:0/2415299807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7441a0dd0 con 0x7ff744106000 2024-04-26T20:10:23.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff74a600700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff74410b640 0x7ff7441a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:10:23.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff749200700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff744106000 0x7ff74419c300 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff730009230 tx=0x7ff7300094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:23.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff742000700 1 -- 172.21.15.94:0/2415299807 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730005220 con 0x7ff744106000 2024-04-26T20:10:23.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff742000700 1 -- 172.21.15.94:0/2415299807 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff73000b560 con 0x7ff744106000 2024-04-26T20:10:23.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7441a1060 con 0x7ff744106000 2024-04-26T20:10:23.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.583+0000 7ff742000700 1 -- 172.21.15.94:0/2415299807 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730003b50 con 0x7ff744106000 2024-04-26T20:10:23.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.584+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7441a8940 con 0x7ff744106000 2024-04-26T20:10:23.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.584+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff744063bb0 con 0x7ff744106000 2024-04-26T20:10:23.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.587+0000 7ff742000700 1 -- 172.21.15.94:0/2415299807 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff73000b710 con 0x7ff744106000 2024-04-26T20:10:23.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.588+0000 7ff742000700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff72c0a3d50 0x7ff72c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:23.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.588+0000 7ff742000700 1 -- 172.21.15.94:0/2415299807 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7ff7300d1e50 con 0x7ff744106000 2024-04-26T20:10:23.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.588+0000 7ff749c00700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff72c0a3d50 0x7ff72c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:23.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.589+0000 7ff742000700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] conn(0x7ff72c0aef80 0x7ff72c0b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:23.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.589+0000 7ff742000700 1 -- 172.21.15.94:0/2415299807 --> [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff730001d30 con 0x7ff72c0aef80 2024-04-26T20:10:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.589+0000 7ff742000700 1 -- 172.21.15.94:0/2415299807 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7ff7300d2200 con 0x7ff744106000 2024-04-26T20:10:23.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.589+0000 7ff74a600700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] conn(0x7ff72c0aef80 0x7ff72c0b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:23.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.589+0000 7ff749c00700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff72c0a3d50 0x7ff72c0a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff738009750 tx=0x7ff738009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:23.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.590+0000 7ff74a600700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] conn(0x7ff72c0aef80 0x7ff72c0b1420 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:23.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.591+0000 7ff742000700 1 -- 172.21.15.94:0/2415299807 <== osd.10 v2:172.21.15.103:6824/2183890761 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff730001d30 con 0x7ff72c0aef80 2024-04-26T20:10:23.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.609+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 --> [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff744063fb0 con 0x7ff72c0aef80 2024-04-26T20:10:23.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.610+0000 7ff742000700 1 -- 172.21.15.94:0/2415299807 <== osd.10 v2:172.21.15.103:6824/2183890761 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff744063fb0 con 0x7ff72c0aef80 2024-04-26T20:10:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 >> [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] conn(0x7ff72c0aef80 msgr2=0x7ff72c0b1420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:23.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] conn(0x7ff72c0aef80 0x7ff72c0b1420 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:23.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff72c0a3d50 msgr2=0x7ff72c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:23.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff72c0a3d50 0x7ff72c0a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff738009750 tx=0x7ff738009310 comp rx=0 tx=0).stop 2024-04-26T20:10:24.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff744106000 msgr2=0x7ff74419c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:24.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 --2- 172.21.15.94:0/2415299807 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff744106000 0x7ff74419c300 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff730009230 tx=0x7ff7300094a0 comp rx=0 tx=0).stop 2024-04-26T20:10:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74a600700 1 -- 172.21.15.94:0/2415299807 reap_dead start 2024-04-26T20:10:24.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 shutdown_connections 2024-04-26T20:10:24.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 >> 172.21.15.94:0/2415299807 conn(0x7ff744101130 msgr2=0x7ff744101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:24.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 shutdown_connections 2024-04-26T20:10:24.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.611+0000 7ff74c600700 1 -- 172.21.15.94:0/2415299807 wait complete. 2024-04-26T20:10:24.168 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:22 smithi094 ceph-mon[28050]: pgmap v391: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 180 B/s, 1 objects/s recovering 2024-04-26T20:10:24.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:22 smithi204 ceph-mon[32618]: pgmap v391: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 180 B/s, 1 objects/s recovering 2024-04-26T20:10:24.183 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:22 smithi103 ceph-mon[32635]: pgmap v391: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 180 B/s, 1 objects/s recovering 2024-04-26T20:10:24.190 INFO:teuthology.orchestra.run.smithi094.stdout:219043332158 2024-04-26T20:10:24.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T20:10:24.437 INFO:teuthology.orchestra.run.smithi094.stdout:55834574945 2024-04-26T20:10:24.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T20:10:24.934 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:24 smithi204 ceph-mon[32618]: pgmap v392: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 159 B/s, 0 objects/s recovering 2024-04-26T20:10:24.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:24 smithi094 ceph-mon[28050]: pgmap v392: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 159 B/s, 0 objects/s recovering 2024-04-26T20:10:25.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:24 smithi103 ceph-mon[32635]: pgmap v392: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 159 B/s, 0 objects/s recovering 2024-04-26T20:10:25.075 INFO:teuthology.orchestra.run.smithi094.stdout:34359738467 2024-04-26T20:10:25.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T20:10:25.211 INFO:teuthology.orchestra.run.smithi094.stdout:287762808883 2024-04-26T20:10:25.222 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T20:10:25.289 INFO:teuthology.orchestra.run.smithi094.stdout:343597383720 2024-04-26T20:10:25.298 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T20:10:25.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.462+0000 7f2b9bc00700 1 -- 172.21.15.94:0/4263673413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b94103470 msgr2=0x7f2b941038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:25.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.462+0000 7f2b9bc00700 1 --2- 172.21.15.94:0/4263673413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b94103470 0x7f2b941038f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000faf0 tx=0x7f2b8000fe00 comp rx=0 tx=0).stop 2024-04-26T20:10:25.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.462+0000 7f2b9bc00700 1 -- 172.21.15.94:0/4263673413 shutdown_connections 2024-04-26T20:10:25.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.462+0000 7f2b9bc00700 1 --2- 172.21.15.94:0/4263673413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b94103e30 0x7f2b94110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:25.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.462+0000 7f2b9bc00700 1 --2- 172.21.15.94:0/4263673413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b941094c0 0x7f2b941098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.462+0000 7f2b9bc00700 1 --2- 172.21.15.94:0/4263673413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b94103470 0x7f2b941038f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.463+0000 7f2b9bc00700 1 -- 172.21.15.94:0/4263673413 >> 172.21.15.94:0/4263673413 conn(0x7f2b9407a580 msgr2=0x7f2b9407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:25.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.464+0000 7f2b9bc00700 1 -- 172.21.15.94:0/4263673413 shutdown_connections 2024-04-26T20:10:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.464+0000 7f2b9bc00700 1 -- 172.21.15.94:0/4263673413 wait complete. 2024-04-26T20:10:25.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.465+0000 7f2b9bc00700 1 Processor -- start 2024-04-26T20:10:25.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.466+0000 7f2b9bc00700 1 -- start start 2024-04-26T20:10:25.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.466+0000 7f2b9bc00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b94103e30 0x7f2b941a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:25.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.466+0000 7f2b9bc00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b941094c0 0x7f2b941a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:25.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.466+0000 7f2b9bc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b941a4c70 0x7f2b941a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:25.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.466+0000 7f2b9bc00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2b94119c60 con 0x7f2b941094c0 2024-04-26T20:10:25.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.466+0000 7f2b9bc00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b94119ae0 con 0x7f2b941a4c70 2024-04-26T20:10:25.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.466+0000 7f2b9bc00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2b94119de0 con 0x7f2b94103e30 2024-04-26T20:10:25.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b94103e30 0x7f2b941a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:25.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b94103e30 0x7f2b941a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:48886/0 (socket says 172.21.15.94:48886) 2024-04-26T20:10:25.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99200700 1 -- 172.21.15.94:0/3460208665 learned_addr learned my addr 172.21.15.94:0/3460208665 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:25.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b93e00700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b941094c0 0x7f2b941a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:25.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99c00700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b941a4c70 0x7f2b941a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:25.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99200700 1 -- 172.21.15.94:0/3460208665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b941a4c70 msgr2=0x7f2b941a50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:25.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99200700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b941a4c70 0x7f2b941a50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:25.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99200700 1 -- 172.21.15.94:0/3460208665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b941094c0 msgr2=0x7f2b941a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:25.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99200700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b941094c0 0x7f2b941a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99200700 1 -- 172.21.15.94:0/3460208665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b941a5790 con 0x7f2b94103e30 2024-04-26T20:10:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.467+0000 7f2b99200700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b94103e30 0x7f2b941a03a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800e9a0 tx=0x7f2b8800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.468+0000 7f2b91600700 1 -- 172.21.15.94:0/3460208665 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8800cbd0 con 0x7f2b94103e30 2024-04-26T20:10:26.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.468+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b941a5a80 con 0x7f2b94103e30 2024-04-26T20:10:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.468+0000 7f2b91600700 1 -- 172.21.15.94:0/3460208665 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b8800cd70 con 0x7f2b94103e30 2024-04-26T20:10:26.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.468+0000 7f2b91600700 1 -- 172.21.15.94:0/3460208665 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b88018680 con 0x7f2b94103e30 2024-04-26T20:10:26.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.468+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b941ad200 con 0x7f2b94103e30 2024-04-26T20:10:26.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.469+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2b94063ab0 con 0x7f2b94103e30 2024-04-26T20:10:26.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.471+0000 7f2b91600700 1 -- 172.21.15.94:0/3460208665 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2b88010430 con 0x7f2b94103e30 2024-04-26T20:10:26.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.473+0000 7f2b91600700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2b7c0a3cd0 0x7f2b7c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:26.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.473+0000 7f2b91600700 1 -- 172.21.15.94:0/3460208665 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f2b880d0ff0 con 0x7f2b94103e30 2024-04-26T20:10:26.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.473+0000 7f2b93e00700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2b7c0a3cd0 0x7f2b7c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.473+0000 7f2b91600700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] conn(0x7f2b7c0af030 0x7f2b7c0b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.473+0000 7f2b91600700 1 -- 172.21.15.94:0/3460208665 --> [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b7c0b1ba0 con 0x7f2b7c0af030 2024-04-26T20:10:26.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.473+0000 7f2b91600700 1 -- 172.21.15.94:0/3460208665 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f2b880d1440 con 0x7f2b94103e30 2024-04-26T20:10:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.474+0000 7f2b99c00700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] conn(0x7f2b7c0af030 0x7f2b7c0b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:26.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.474+0000 7f2b93e00700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2b7c0a3cd0 0x7f2b7c0a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009fd0 tx=0x7f2b8000ff90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.475+0000 7f2b99c00700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] conn(0x7f2b7c0af030 0x7f2b7c0b14d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.476+0000 7f2b91600700 1 -- 172.21.15.94:0/3460208665 <== osd.4 v2:172.21.15.103:6808/3296783557 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2b7c0b1ba0 con 0x7f2b7c0af030 2024-04-26T20:10:26.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.514+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 --> [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2b94063f30 con 0x7f2b7c0af030 2024-04-26T20:10:26.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.514+0000 7f2b91600700 1 -- 172.21.15.94:0/3460208665 <== osd.4 v2:172.21.15.103:6808/3296783557 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2b94063f30 con 0x7f2b7c0af030 2024-04-26T20:10:26.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.515+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 >> [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] conn(0x7f2b7c0af030 msgr2=0x7f2b7c0b14d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:26.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.515+0000 7f2b9bc00700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] conn(0x7f2b7c0af030 0x7f2b7c0b14d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:26.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.515+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2b7c0a3cd0 msgr2=0x7f2b7c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:26.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.515+0000 7f2b9bc00700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2b7c0a3cd0 0x7f2b7c0a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009fd0 tx=0x7f2b8000ff90 comp rx=0 tx=0).stop 2024-04-26T20:10:26.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.515+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b94103e30 msgr2=0x7f2b941a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:26.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.515+0000 7f2b9bc00700 1 --2- 172.21.15.94:0/3460208665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b94103e30 0x7f2b941a03a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800e9a0 tx=0x7f2b8800ecb0 comp rx=0 tx=0).stop 2024-04-26T20:10:26.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.515+0000 7f2b99c00700 1 -- 172.21.15.94:0/3460208665 reap_dead start 2024-04-26T20:10:26.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.516+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 shutdown_connections 2024-04-26T20:10:26.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.516+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 >> 172.21.15.94:0/3460208665 conn(0x7f2b9407a580 msgr2=0x7f2b941023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:26.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.516+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 shutdown_connections 2024-04-26T20:10:26.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:25.516+0000 7f2b9bc00700 1 -- 172.21.15.94:0/3460208665 wait complete. 2024-04-26T20:10:26.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.058+0000 7fa998c00700 1 -- 172.21.15.94:0/626376349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9941034d0 msgr2=0x7fa994103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:26.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.058+0000 7fa998c00700 1 --2- 172.21.15.94:0/626376349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9941034d0 0x7fa994103950 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa9800099c0 tx=0x7fa980009cd0 comp rx=0 tx=0).stop 2024-04-26T20:10:26.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.058+0000 7fa998c00700 1 -- 172.21.15.94:0/626376349 shutdown_connections 2024-04-26T20:10:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.058+0000 7fa998c00700 1 --2- 172.21.15.94:0/626376349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa994103e90 0x7fa994110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:26.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.058+0000 7fa998c00700 1 --2- 172.21.15.94:0/626376349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9941083b0 0x7fa994108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:26.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.058+0000 7fa998c00700 1 --2- 172.21.15.94:0/626376349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9941034d0 0x7fa994103950 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:26.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.058+0000 7fa998c00700 1 -- 172.21.15.94:0/626376349 >> 172.21.15.94:0/626376349 conn(0x7fa99407a580 msgr2=0x7fa99407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:26.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.058+0000 7fa998c00700 1 -- 172.21.15.94:0/626376349 shutdown_connections 2024-04-26T20:10:26.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.058+0000 7fa998c00700 1 -- 172.21.15.94:0/626376349 wait complete. 2024-04-26T20:10:26.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.059+0000 7fa998c00700 1 Processor -- start 2024-04-26T20:10:26.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.059+0000 7fa998c00700 1 -- start start 2024-04-26T20:10:27.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa998c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9941034d0 0x7fa99419bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:27.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa998c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa994103e90 0x7fa99419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:27.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa998c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9941083b0 0x7fa9941a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:27.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa998c00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa994113690 con 0x7fa9941083b0 2024-04-26T20:10:27.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa998c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa994113510 con 0x7fa994103e90 2024-04-26T20:10:27.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa998c00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa994113810 con 0x7fa9941034d0 2024-04-26T20:10:27.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa991e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9941034d0 0x7fa99419bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:27.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa992800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9941083b0 0x7fa9941a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:27.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa991e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9941034d0 0x7fa99419bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:48856/0 (socket says 172.21.15.94:48856) 2024-04-26T20:10:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa991e00700 1 -- 172.21.15.94:0/3200358239 learned_addr learned my addr 172.21.15.94:0/3200358239 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:27.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa991400700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa994103e90 0x7fa99419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:27.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa992800700 1 -- 172.21.15.94:0/3200358239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9941034d0 msgr2=0x7fa99419bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa992800700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9941034d0 0x7fa99419bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:27.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa992800700 1 -- 172.21.15.94:0/3200358239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa994103e90 msgr2=0x7fa99419c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa992800700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa994103e90 0x7fa99419c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa992800700 1 -- 172.21.15.94:0/3200358239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9941a0d50 con 0x7fa9941083b0 2024-04-26T20:10:27.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.060+0000 7fa991e00700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9941034d0 0x7fa99419bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:10:27.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.061+0000 7fa992800700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9941083b0 0x7fa9941a0610 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00efc0 tx=0x7fa98c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:27.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.061+0000 7fa98aa00700 1 -- 172.21.15.94:0/3200358239 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0092e0 con 0x7fa9941083b0 2024-04-26T20:10:27.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.061+0000 7fa98aa00700 1 -- 172.21.15.94:0/3200358239 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa98c004bb0 con 0x7fa9941083b0 2024-04-26T20:10:27.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.061+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9941a0fe0 con 0x7fa9941083b0 2024-04-26T20:10:27.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.061+0000 7fa98aa00700 1 -- 172.21.15.94:0/3200358239 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c004020 con 0x7fa9941083b0 2024-04-26T20:10:27.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.061+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9941a88c0 con 0x7fa9941083b0 2024-04-26T20:10:27.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.062+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa974001030 con 0x7fa9941083b0 2024-04-26T20:10:27.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.066+0000 7fa98aa00700 1 -- 172.21.15.94:0/3200358239 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa98c01b030 con 0x7fa9941083b0 2024-04-26T20:10:27.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.067+0000 7fa98aa00700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9780a3cd0 0x7fa9780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:27.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.068+0000 7fa98aa00700 1 -- 172.21.15.94:0/3200358239 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fa98c0d0ce0 con 0x7fa9941083b0 2024-04-26T20:10:27.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.068+0000 7fa991e00700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9780a3cd0 0x7fa9780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:27.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.068+0000 7fa98aa00700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] conn(0x7fa9780af030 0x7fa9780b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:27.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.068+0000 7fa98aa00700 1 -- 172.21.15.94:0/3200358239 --> [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9780b1ba0 con 0x7fa9780af030 2024-04-26T20:10:27.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.068+0000 7fa98aa00700 1 -- 172.21.15.94:0/3200358239 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fa98c0d1090 con 0x7fa9941083b0 2024-04-26T20:10:27.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.068+0000 7fa991400700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] conn(0x7fa9780af030 0x7fa9780b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:27.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.069+0000 7fa991e00700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9780a3cd0 0x7fa9780a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa984011550 tx=0x7fa984009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:27.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.069+0000 7fa991400700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] conn(0x7fa9780af030 0x7fa9780b14d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:27.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.070+0000 7fa98aa00700 1 -- 172.21.15.94:0/3200358239 <== osd.3 v2:172.21.15.94:6810/2184382995 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa9780b1ba0 con 0x7fa9780af030 2024-04-26T20:10:27.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.089+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 --> [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa974002e60 con 0x7fa9780af030 2024-04-26T20:10:27.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.090+0000 7fa98aa00700 1 -- 172.21.15.94:0/3200358239 <== osd.3 v2:172.21.15.94:6810/2184382995 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa974002e60 con 0x7fa9780af030 2024-04-26T20:10:27.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.090+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] conn(0x7fa9780af030 msgr2=0x7fa9780b14d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:27.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.090+0000 7fa998c00700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] conn(0x7fa9780af030 0x7fa9780b14d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:27.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.090+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9780a3cd0 msgr2=0x7fa9780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.091+0000 7fa998c00700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa9780a3cd0 0x7fa9780a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa984011550 tx=0x7fa984009260 comp rx=0 tx=0).stop 2024-04-26T20:10:28.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.091+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9941083b0 msgr2=0x7fa9941a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:28.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.091+0000 7fa998c00700 1 --2- 172.21.15.94:0/3200358239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9941083b0 0x7fa9941a0610 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00efc0 tx=0x7fa98c00c3a0 comp rx=0 tx=0).stop 2024-04-26T20:10:28.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.091+0000 7fa992800700 1 -- 172.21.15.94:0/3200358239 reap_dead start 2024-04-26T20:10:28.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.091+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 shutdown_connections 2024-04-26T20:10:28.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.091+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 >> 172.21.15.94:0/3200358239 conn(0x7fa99407a580 msgr2=0x7fa994101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:28.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.091+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 shutdown_connections 2024-04-26T20:10:28.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:22.091+0000 7fa998c00700 1 -- 172.21.15.94:0/3200358239 wait complete. 2024-04-26T20:10:28.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:26 smithi204 ceph-mon[32618]: pgmap v393: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 134 B/s, 0 objects/s recovering 2024-04-26T20:10:28.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:26 smithi103 ceph-mon[32635]: pgmap v393: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 134 B/s, 0 objects/s recovering 2024-04-26T20:10:28.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.533+0000 7f2156800700 1 -- 172.21.15.94:0/2896849482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f215006bad0 msgr2=0x7f21501102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:28.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.533+0000 7f2156800700 1 --2- 172.21.15.94:0/2896849482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f215006bad0 0x7f21501102e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2148009990 tx=0x7f2148009ca0 comp rx=0 tx=0).stop 2024-04-26T20:10:28.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.533+0000 7f2156800700 1 -- 172.21.15.94:0/2896849482 shutdown_connections 2024-04-26T20:10:28.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.533+0000 7f2156800700 1 --2- 172.21.15.94:0/2896849482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f215006bad0 0x7f21501102e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:28.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.533+0000 7f2156800700 1 --2- 172.21.15.94:0/2896849482 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21501094a0 0x7f2150109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:28.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.533+0000 7f2156800700 1 --2- 172.21.15.94:0/2896849482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f215006b130 0x7f215006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:28.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.533+0000 7f2156800700 1 -- 172.21.15.94:0/2896849482 >> 172.21.15.94:0/2896849482 conn(0x7f21500792d0 msgr2=0x7f21500796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:28.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.533+0000 7f2156800700 1 -- 172.21.15.94:0/2896849482 shutdown_connections 2024-04-26T20:10:28.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.533+0000 7f2156800700 1 -- 172.21.15.94:0/2896849482 wait complete. 2024-04-26T20:10:28.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.534+0000 7f2156800700 1 Processor -- start 2024-04-26T20:10:28.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.534+0000 7f2156800700 1 -- start start 2024-04-26T20:10:28.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f2156800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215006b130 0x7f215019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:28.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f2156800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f215006bad0 0x7f215019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:28.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f2156800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21501094a0 0x7f2150199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:28.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f2156800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f21501135f0 con 0x7f215006b130 2024-04-26T20:10:28.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f2156800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2150113470 con 0x7f21501094a0 2024-04-26T20:10:28.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f2156800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2150113770 con 0x7f215006bad0 2024-04-26T20:10:28.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f214f400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215006b130 0x7f215019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f214f400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215006b130 0x7f215019f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42830/0 (socket says 172.21.15.94:42830) 2024-04-26T20:10:28.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f214f400700 1 -- 172.21.15.94:0/1949674321 learned_addr learned my addr 172.21.15.94:0/1949674321 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:28.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f2146a00700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f215006bad0 0x7f215019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:28.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f214fe00700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21501094a0 0x7f2150199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:28.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f214f400700 1 -- 172.21.15.94:0/1949674321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f215006bad0 msgr2=0x7f215019fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:28.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f214f400700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f215006bad0 0x7f215019fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:28.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f214f400700 1 -- 172.21.15.94:0/1949674321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21501094a0 msgr2=0x7f2150199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:28.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f214f400700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21501094a0 0x7f2150199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:28.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.535+0000 7f214f400700 1 -- 172.21.15.94:0/1949674321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215019a090 con 0x7f215006b130 2024-04-26T20:10:29.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.536+0000 7f214f400700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215006b130 0x7f215019f770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f214000d980 tx=0x7f214000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:29.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.536+0000 7f214cc00700 1 -- 172.21.15.94:0/1949674321 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140016070 con 0x7f215006b130 2024-04-26T20:10:29.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.536+0000 7f214cc00700 1 -- 172.21.15.94:0/1949674321 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2140004830 con 0x7f215006b130 2024-04-26T20:10:29.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.536+0000 7f214cc00700 1 -- 172.21.15.94:0/1949674321 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140005540 con 0x7f215006b130 2024-04-26T20:10:29.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.536+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215019a380 con 0x7f215006b130 2024-04-26T20:10:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.536+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21501a6640 con 0x7f215006b130 2024-04-26T20:10:29.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.537+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2150063b70 con 0x7f215006b130 2024-04-26T20:10:29.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.539+0000 7f214cc00700 1 -- 172.21.15.94:0/1949674321 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2140004220 con 0x7f215006b130 2024-04-26T20:10:29.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.541+0000 7f214cc00700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21340a3d50 0x7f21340a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:29.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.541+0000 7f214cc00700 1 -- 172.21.15.94:0/1949674321 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f21400d0d30 con 0x7f215006b130 2024-04-26T20:10:29.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.541+0000 7f2146a00700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21340a3d50 0x7f21340a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:29.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.541+0000 7f214cc00700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] conn(0x7f21340af0b0 0x7f21340b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:29.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.541+0000 7f214cc00700 1 -- 172.21.15.94:0/1949674321 --> [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2140015ec0 con 0x7f21340af0b0 2024-04-26T20:10:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.541+0000 7f214cc00700 1 -- 172.21.15.94:0/1949674321 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f21400d10e0 con 0x7f215006b130 2024-04-26T20:10:29.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.541+0000 7f214fe00700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] conn(0x7f21340af0b0 0x7f21340b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.542+0000 7f2146a00700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21340a3d50 0x7f21340a6210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f21500799b0 tx=0x7f213800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:29.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.542+0000 7f214fe00700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] conn(0x7f21340af0b0 0x7f21340b1550 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.543+0000 7f214cc00700 1 -- 172.21.15.94:0/1949674321 <== osd.6 v2:172.21.15.94:6818/4225405603 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2140015ec0 con 0x7f21340af0b0 2024-04-26T20:10:29.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.561+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 --> [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2150063fa0 con 0x7f21340af0b0 2024-04-26T20:10:29.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.562+0000 7f214cc00700 1 -- 172.21.15.94:0/1949674321 <== osd.6 v2:172.21.15.94:6818/4225405603 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2150063fa0 con 0x7f21340af0b0 2024-04-26T20:10:29.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.562+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] conn(0x7f21340af0b0 msgr2=0x7f21340b1550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:29.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.562+0000 7f2156800700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] conn(0x7f21340af0b0 0x7f21340b1550 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:29.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.563+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21340a3d50 msgr2=0x7f21340a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:29.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.563+0000 7f2156800700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21340a3d50 0x7f21340a6210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f21500799b0 tx=0x7f213800b040 comp rx=0 tx=0).stop 2024-04-26T20:10:29.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.563+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215006b130 msgr2=0x7f215019f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:29.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.563+0000 7f2156800700 1 --2- 172.21.15.94:0/1949674321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f215006b130 0x7f215019f770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f214000d980 tx=0x7f214000dc90 comp rx=0 tx=0).stop 2024-04-26T20:10:29.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.563+0000 7f214fe00700 1 -- 172.21.15.94:0/1949674321 reap_dead start 2024-04-26T20:10:29.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.563+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 shutdown_connections 2024-04-26T20:10:29.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.563+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 >> 172.21.15.94:0/1949674321 conn(0x7f21500792d0 msgr2=0x7f2150100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:29.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.563+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 shutdown_connections 2024-04-26T20:10:29.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:26.563+0000 7f2156800700 1 -- 172.21.15.94:0/1949674321 wait complete. 2024-04-26T20:10:29.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:26 smithi094 ceph-mon[28050]: pgmap v393: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 134 B/s, 0 objects/s recovering 2024-04-26T20:10:29.891 INFO:teuthology.orchestra.run.smithi094.stdout:107374182486 2024-04-26T20:10:29.923 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T20:10:29.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.341+0000 7f2202600700 1 -- 172.21.15.94:0/1280796967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21fc10b640 msgr2=0x7f21fc10ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:29.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.341+0000 7f2202600700 1 --2- 172.21.15.94:0/1280796967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21fc10b640 0x7f21fc10ba20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009990 tx=0x7f21ec009ca0 comp rx=0 tx=0).stop 2024-04-26T20:10:29.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.341+0000 7f2202600700 1 -- 172.21.15.94:0/1280796967 shutdown_connections 2024-04-26T20:10:29.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.341+0000 7f2202600700 1 --2- 172.21.15.94:0/1280796967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21fc106000 0x7f21fc112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:30.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.341+0000 7f2202600700 1 --2- 172.21.15.94:0/1280796967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21fc10b640 0x7f21fc10ba20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:30.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.341+0000 7f2202600700 1 --2- 172.21.15.94:0/1280796967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21fc105640 0x7f21fc105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:30.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.341+0000 7f2202600700 1 -- 172.21.15.94:0/1280796967 >> 172.21.15.94:0/1280796967 conn(0x7f21fc101130 msgr2=0x7f21fc103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:30.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.341+0000 7f2202600700 1 -- 172.21.15.94:0/1280796967 shutdown_connections 2024-04-26T20:10:30.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.341+0000 7f2202600700 1 -- 172.21.15.94:0/1280796967 wait complete. 2024-04-26T20:10:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.342+0000 7f2202600700 1 Processor -- start 2024-04-26T20:10:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.342+0000 7f2202600700 1 -- start start 2024-04-26T20:10:30.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f2202600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21fc105640 0x7f21fc1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:30.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f2202600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21fc106000 0x7f21fc1a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f2202600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21fc10b640 0x7f21fc19bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:30.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f2202600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc115870 con 0x7f21fc106000 2024-04-26T20:10:30.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f2202600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1156f0 con 0x7f21fc105640 2024-04-26T20:10:30.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f2202600700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1159f0 con 0x7f21fc10b640 2024-04-26T20:10:30.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f21f2a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21fc106000 0x7f21fc1a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:30.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f21f2a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21fc106000 0x7f21fc1a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42850/0 (socket says 172.21.15.94:42850) 2024-04-26T20:10:30.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f21f2a00700 1 -- 172.21.15.94:0/2868466805 learned_addr learned my addr 172.21.15.94:0/2868466805 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:30.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f21fbe00700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21fc10b640 0x7f21fc19bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:30.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.343+0000 7f21fb400700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21fc105640 0x7f21fc1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:30.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f21f2a00700 1 -- 172.21.15.94:0/2868466805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21fc10b640 msgr2=0x7f21fc19bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:30.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f21f2a00700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21fc10b640 0x7f21fc19bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:30.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f21f2a00700 1 -- 172.21.15.94:0/2868466805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21fc105640 msgr2=0x7f21fc1a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:30.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f21f2a00700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21fc105640 0x7f21fc1a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:30.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f21f2a00700 1 -- 172.21.15.94:0/2868466805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21fc19c2b0 con 0x7f21fc106000 2024-04-26T20:10:30.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f21f2a00700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21fc106000 0x7f21fc1a1e80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f21e400e9a0 tx=0x7f21e400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:30.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f21f8c00700 1 -- 172.21.15.94:0/2868466805 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e400ca10 con 0x7f21fc106000 2024-04-26T20:10:30.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f21f8c00700 1 -- 172.21.15.94:0/2868466805 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21e400cbb0 con 0x7f21fc106000 2024-04-26T20:10:30.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21fc19c5a0 con 0x7f21fc106000 2024-04-26T20:10:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f21f8c00700 1 -- 172.21.15.94:0/2868466805 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4018680 con 0x7f21fc106000 2024-04-26T20:10:30.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.344+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21fc1a8810 con 0x7f21fc106000 2024-04-26T20:10:30.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.345+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f21fc063bb0 con 0x7f21fc106000 2024-04-26T20:10:30.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.348+0000 7f21f8c00700 1 -- 172.21.15.94:0/2868466805 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f21e4010430 con 0x7f21fc106000 2024-04-26T20:10:30.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.349+0000 7f21f8c00700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21e00a3d50 0x7f21e00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:30.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.349+0000 7f21f8c00700 1 -- 172.21.15.94:0/2868466805 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f21e40d0ff0 con 0x7f21fc106000 2024-04-26T20:10:30.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.349+0000 7f21fb400700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21e00a3d50 0x7f21e00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:30.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.350+0000 7f21f8c00700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] conn(0x7f21e00af0b0 0x7f21e00b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:30.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.350+0000 7f21f8c00700 1 -- 172.21.15.94:0/2868466805 --> [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21e4019ec0 con 0x7f21e00af0b0 2024-04-26T20:10:30.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.350+0000 7f21f8c00700 1 -- 172.21.15.94:0/2868466805 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f21e40d1420 con 0x7f21fc106000 2024-04-26T20:10:30.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.350+0000 7f21fb400700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21e00a3d50 0x7f21e00a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00b2e0 tx=0x7f21ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:31.055 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:28 smithi204 ceph-mon[32618]: pgmap v394: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 132 B/s, 0 objects/s recovering 2024-04-26T20:10:31.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:28 smithi103 ceph-mon[32635]: pgmap v394: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 132 B/s, 0 objects/s recovering 2024-04-26T20:10:31.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.350+0000 7f21fbe00700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] conn(0x7f21e00af0b0 0x7f21e00b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:31.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.351+0000 7f21fbe00700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] conn(0x7f21e00af0b0 0x7f21e00b1550 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:31.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.352+0000 7f21f8c00700 1 -- 172.21.15.94:0/2868466805 <== osd.5 v2:172.21.15.204:6808/1603790096 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f21e4019ec0 con 0x7f21e00af0b0 2024-04-26T20:10:31.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.370+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 --> [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f21fc063fb0 con 0x7f21e00af0b0 2024-04-26T20:10:31.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.370+0000 7f21f8c00700 1 -- 172.21.15.94:0/2868466805 <== osd.5 v2:172.21.15.204:6808/1603790096 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f21fc063fb0 con 0x7f21e00af0b0 2024-04-26T20:10:31.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 >> [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] conn(0x7f21e00af0b0 msgr2=0x7f21e00b1550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:31.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] conn(0x7f21e00af0b0 0x7f21e00b1550 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:31.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21e00a3d50 msgr2=0x7f21e00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f21e00a3d50 0x7f21e00a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00b2e0 tx=0x7f21ec005fd0 comp rx=0 tx=0).stop 2024-04-26T20:10:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21fc106000 msgr2=0x7f21fc1a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:31.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 --2- 172.21.15.94:0/2868466805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21fc106000 0x7f21fc1a1e80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f21e400e9a0 tx=0x7f21e400ecb0 comp rx=0 tx=0).stop 2024-04-26T20:10:31.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f21fbe00700 1 -- 172.21.15.94:0/2868466805 reap_dead start 2024-04-26T20:10:31.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 shutdown_connections 2024-04-26T20:10:31.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 >> 172.21.15.94:0/2868466805 conn(0x7f21fc101130 msgr2=0x7f21fc101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 shutdown_connections 2024-04-26T20:10:31.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:28.371+0000 7f2202600700 1 -- 172.21.15.94:0/2868466805 wait complete. 2024-04-26T20:10:31.475 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:28 smithi094 ceph-mon[28050]: pgmap v394: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 132 B/s, 0 objects/s recovering 2024-04-26T20:10:31.483 INFO:teuthology.orchestra.run.smithi094.stdout:141733920849 2024-04-26T20:10:31.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T20:10:31.498 INFO:teuthology.orchestra.run.smithi094.stdout:309237645357 2024-04-26T20:10:31.506 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T20:10:31.513 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:30 smithi204 ceph-mon[32618]: pgmap v395: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 84 B/s, 0 objects/s recovering 2024-04-26T20:10:31.520 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:30 smithi103 ceph-mon[32635]: pgmap v395: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 84 B/s, 0 objects/s recovering 2024-04-26T20:10:31.565 INFO:teuthology.orchestra.run.smithi094.stdout:163208757325 2024-04-26T20:10:31.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T20:10:31.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.819+0000 7f2fab200700 1 -- 172.21.15.94:0/4065053711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fa4105570 msgr2=0x7f2fa41059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:31.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.819+0000 7f2fab200700 1 --2- 172.21.15.94:0/4065053711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fa4105570 0x7f2fa41059f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2f940099c0 tx=0x7f2f94009cd0 comp rx=0 tx=0).stop 2024-04-26T20:10:31.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.819+0000 7f2fab200700 1 -- 172.21.15.94:0/4065053711 shutdown_connections 2024-04-26T20:10:31.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.819+0000 7f2fab200700 1 --2- 172.21.15.94:0/4065053711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa4105f30 0x7f2fa41123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:31.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.819+0000 7f2fab200700 1 --2- 172.21.15.94:0/4065053711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fa410b620 0x7f2fa410ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:31.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.819+0000 7f2fab200700 1 --2- 172.21.15.94:0/4065053711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fa4105570 0x7f2fa41059f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:31.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.819+0000 7f2fab200700 1 -- 172.21.15.94:0/4065053711 >> 172.21.15.94:0/4065053711 conn(0x7f2fa4101040 msgr2=0x7f2fa4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:31.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.819+0000 7f2fab200700 1 -- 172.21.15.94:0/4065053711 shutdown_connections 2024-04-26T20:10:31.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.820+0000 7f2fab200700 1 -- 172.21.15.94:0/4065053711 wait complete. 2024-04-26T20:10:31.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.820+0000 7f2fab200700 1 Processor -- start 2024-04-26T20:10:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2fab200700 1 -- start start 2024-04-26T20:10:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2fab200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa4105570 0x7f2fa41a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:31.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2fab200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fa4105f30 0x7f2fa41a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:31.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2fab200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fa410b620 0x7f2fa419b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:31.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2fab200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa41157d0 con 0x7f2fa410b620 2024-04-26T20:10:31.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2fab200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa4115650 con 0x7f2fa4105570 2024-04-26T20:10:31.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2fab200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa4115950 con 0x7f2fa4105f30 2024-04-26T20:10:31.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2faa800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fa410b620 0x7f2fa419b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:32.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2faa800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fa410b620 0x7f2fa419b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42860/0 (socket says 172.21.15.94:42860) 2024-04-26T20:10:32.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2faa800700 1 -- 172.21.15.94:0/3808464582 learned_addr learned my addr 172.21.15.94:0/3808464582 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:32.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2fa9e00700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa4105570 0x7f2fa41a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:32.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.821+0000 7f2fa9400700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fa4105f30 0x7f2fa41a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:32.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2faa800700 1 -- 172.21.15.94:0/3808464582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fa4105f30 msgr2=0x7f2fa41a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:32.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2faa800700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fa4105f30 0x7f2fa41a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:32.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2faa800700 1 -- 172.21.15.94:0/3808464582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa4105570 msgr2=0x7f2fa41a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:32.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2faa800700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa4105570 0x7f2fa41a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:32.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2faa800700 1 -- 172.21.15.94:0/3808464582 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa419c150 con 0x7f2fa410b620 2024-04-26T20:10:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2faa800700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fa410b620 0x7f2fa419b9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000ef30 tx=0x7f2fa000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:32.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2f9ea00700 1 -- 172.21.15.94:0/3808464582 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00092e0 con 0x7f2fa410b620 2024-04-26T20:10:32.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2f9ea00700 1 -- 172.21.15.94:0/3808464582 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fa0004bb0 con 0x7f2fa410b620 2024-04-26T20:10:32.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa419c3e0 con 0x7f2fa410b620 2024-04-26T20:10:32.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2f9ea00700 1 -- 172.21.15.94:0/3808464582 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0004020 con 0x7f2fa410b620 2024-04-26T20:10:32.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.822+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa41a86b0 con 0x7f2fa410b620 2024-04-26T20:10:32.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.823+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2fa4063b70 con 0x7f2fa410b620 2024-04-26T20:10:32.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.826+0000 7f2f9ea00700 1 -- 172.21.15.94:0/3808464582 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2fa001b030 con 0x7f2fa410b620 2024-04-26T20:10:32.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.827+0000 7f2f9ea00700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2f8c0a3cd0 0x7f2f8c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:32.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.827+0000 7f2f9ea00700 1 -- 172.21.15.94:0/3808464582 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f2fa00d0e40 con 0x7f2fa410b620 2024-04-26T20:10:32.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.827+0000 7f2fa9e00700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2f8c0a3cd0 0x7f2f8c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:32.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.828+0000 7f2f9ea00700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] conn(0x7f2f8c0af030 0x7f2f8c0b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:32.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.828+0000 7f2f9ea00700 1 -- 172.21.15.94:0/3808464582 --> [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f8c0b1ba0 con 0x7f2f8c0af030 2024-04-26T20:10:32.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.828+0000 7f2f9ea00700 1 -- 172.21.15.94:0/3808464582 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f2fa00d11f0 con 0x7f2fa410b620 2024-04-26T20:10:32.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.828+0000 7f2fa9400700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] conn(0x7f2f8c0af030 0x7f2f8c0b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:32.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.828+0000 7f2fa9e00700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2f8c0a3cd0 0x7f2f8c0a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2fa410a710 tx=0x7f2f98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:32.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.829+0000 7f2fa9400700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] conn(0x7f2f8c0af030 0x7f2f8c0b14d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:32.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.830+0000 7f2f9ea00700 1 -- 172.21.15.94:0/3808464582 <== osd.8 v2:172.21.15.204:6816/1919172767 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2f8c0b1ba0 con 0x7f2f8c0af030 2024-04-26T20:10:32.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.847+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 --> [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2fa4063ff0 con 0x7f2f8c0af030 2024-04-26T20:10:32.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.848+0000 7f2f9ea00700 1 -- 172.21.15.94:0/3808464582 <== osd.8 v2:172.21.15.204:6816/1919172767 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2fa4063ff0 con 0x7f2f8c0af030 2024-04-26T20:10:32.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 >> [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] conn(0x7f2f8c0af030 msgr2=0x7f2f8c0b14d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:32.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] conn(0x7f2f8c0af030 0x7f2f8c0b14d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:32.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2f8c0a3cd0 msgr2=0x7f2f8c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:32.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f2f8c0a3cd0 0x7f2f8c0a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2fa410a710 tx=0x7f2f98009260 comp rx=0 tx=0).stop 2024-04-26T20:10:32.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fa410b620 msgr2=0x7f2fa419b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:32.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 --2- 172.21.15.94:0/3808464582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fa410b620 0x7f2fa419b9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000ef30 tx=0x7f2fa000c3a0 comp rx=0 tx=0).stop 2024-04-26T20:10:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2faa800700 1 -- 172.21.15.94:0/3808464582 reap_dead start 2024-04-26T20:10:32.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 shutdown_connections 2024-04-26T20:10:32.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 >> 172.21.15.94:0/3808464582 conn(0x7f2fa4101040 msgr2=0x7f2fa4102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:32.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 shutdown_connections 2024-04-26T20:10:32.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:29.849+0000 7f2fab200700 1 -- 172.21.15.94:0/3808464582 wait complete. 2024-04-26T20:10:32.959 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:30 smithi094 ceph-mon[28050]: pgmap v395: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 84 B/s, 0 objects/s recovering 2024-04-26T20:10:32.966 INFO:teuthology.orchestra.run.smithi094.stdout:253403070523 2024-04-26T20:10:32.975 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T20:10:32.983 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:32.989 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:32.994 INFO:teuthology.orchestra.run.smithi094.stdout:197568495685 2024-04-26T20:10:32.999 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T20:10:33.006 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:33.047 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:33.075 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:33.081 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:32 smithi094 ceph-mon[28050]: pgmap v396: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:10:33.144 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:32 smithi204 ceph-mon[32618]: pgmap v396: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:10:33.150 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:32 smithi103 ceph-mon[32635]: pgmap v396: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:10:33.697 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:34.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:34 smithi094 ceph-mon[28050]: pgmap v397: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:10:35.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:34 smithi103 ceph-mon[32635]: pgmap v397: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:10:35.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:34 smithi204 ceph-mon[32618]: pgmap v397: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:10:36.836 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:36 smithi094 ceph-mon[28050]: pgmap v398: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:10:37.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:36 smithi103 ceph-mon[32635]: pgmap v398: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:10:37.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:36 smithi204 ceph-mon[32618]: pgmap v398: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:10:37.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.569+0000 7f932da00700 1 -- 172.21.15.94:0/4000533383 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93281035b0 msgr2=0x7f9328103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:37.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.569+0000 7f932da00700 1 --2- 172.21.15.94:0/4000533383 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93281035b0 0x7f9328103990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f93180099c0 tx=0x7f9318009cd0 comp rx=0 tx=0).stop 2024-04-26T20:10:37.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.569+0000 7f932da00700 1 -- 172.21.15.94:0/4000533383 shutdown_connections 2024-04-26T20:10:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.569+0000 7f932da00700 1 --2- 172.21.15.94:0/4000533383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9328108690 0x7f9328110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:37.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.569+0000 7f932da00700 1 --2- 172.21.15.94:0/4000533383 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93281035b0 0x7f9328103990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:37.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.569+0000 7f932da00700 1 --2- 172.21.15.94:0/4000533383 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9328103f60 0x7f9328107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:37.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.569+0000 7f932da00700 1 -- 172.21.15.94:0/4000533383 >> 172.21.15.94:0/4000533383 conn(0x7f93280feea0 msgr2=0x7f93281012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:37.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.569+0000 7f932da00700 1 -- 172.21.15.94:0/4000533383 shutdown_connections 2024-04-26T20:10:37.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.569+0000 7f932da00700 1 -- 172.21.15.94:0/4000533383 wait complete. 2024-04-26T20:10:37.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.570+0000 7f932da00700 1 Processor -- start 2024-04-26T20:10:37.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.570+0000 7f932da00700 1 -- start start 2024-04-26T20:10:37.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f932da00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93281035b0 0x7f93281a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:37.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f932da00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103f60 0x7f93281a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:37.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f932da00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9328108690 0x7f932819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f932da00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9328113590 con 0x7f9328103f60 2024-04-26T20:10:37.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f932da00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9328113410 con 0x7f93281035b0 2024-04-26T20:10:37.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f932da00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9328113710 con 0x7f9328108690 2024-04-26T20:10:37.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103f60 0x7f93281a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103f60 0x7f93281a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49980/0 (socket says 172.21.15.94:49980) 2024-04-26T20:10:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327400700 1 -- 172.21.15.94:0/505145612 learned_addr learned my addr 172.21.15.94:0/505145612 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:37.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327e00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93281035b0 0x7f93281a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:37.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f932d000700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9328108690 0x7f932819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:37.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327400700 1 -- 172.21.15.94:0/505145612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9328108690 msgr2=0x7f932819bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:37.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327400700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9328108690 0x7f932819bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:37.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327400700 1 -- 172.21.15.94:0/505145612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93281035b0 msgr2=0x7f93281a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:37.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327400700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93281035b0 0x7f93281a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:37.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327400700 1 -- 172.21.15.94:0/505145612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932819c410 con 0x7f9328103f60 2024-04-26T20:10:37.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.571+0000 7f9327e00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93281035b0 0x7f93281a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:10:37.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.572+0000 7f9327400700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103f60 0x7f93281a1fa0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f931400e9f0 tx=0x7f931400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:37.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.572+0000 7f9324c00700 1 -- 172.21.15.94:0/505145612 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314004a30 con 0x7f9328103f60 2024-04-26T20:10:38.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.572+0000 7f9324c00700 1 -- 172.21.15.94:0/505145612 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f931400cc70 con 0x7f9328103f60 2024-04-26T20:10:38.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.572+0000 7f9324c00700 1 -- 172.21.15.94:0/505145612 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314005530 con 0x7f9328103f60 2024-04-26T20:10:38.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.572+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932819c700 con 0x7f9328103f60 2024-04-26T20:10:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.572+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93281a89e0 con 0x7f9328103f60 2024-04-26T20:10:38.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.573+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9328104a90 con 0x7f9328103f60 2024-04-26T20:10:38.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.576+0000 7f9324c00700 1 -- 172.21.15.94:0/505145612 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9314004bd0 con 0x7f9328103f60 2024-04-26T20:10:38.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.577+0000 7f9324c00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f930c0a3d50 0x7f930c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:38.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.578+0000 7f9324c00700 1 -- 172.21.15.94:0/505145612 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f93140d0b10 con 0x7f9328103f60 2024-04-26T20:10:38.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.578+0000 7f9327e00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f930c0a3d50 0x7f930c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:38.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.578+0000 7f9327e00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f930c0a3d50 0x7f930c0a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9318003760 tx=0x7f9318005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:38.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.582+0000 7f9324c00700 1 -- 172.21.15.94:0/505145612 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f931400ce20 con 0x7f9328103f60 2024-04-26T20:10:38.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.815+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f932819cff0 con 0x7f9328103f60 2024-04-26T20:10:38.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.816+0000 7f9324c00700 1 -- 172.21.15.94:0/505145612 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f931409cf60 con 0x7f9328103f60 2024-04-26T20:10:38.276 INFO:teuthology.orchestra.run.smithi094.stdout:77309411423 2024-04-26T20:10:38.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.818+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f930c0a3d50 msgr2=0x7f930c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:38.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.818+0000 7f932da00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f930c0a3d50 0x7f930c0a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9318003760 tx=0x7f9318005fb0 comp rx=0 tx=0).stop 2024-04-26T20:10:38.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.818+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103f60 msgr2=0x7f93281a1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:38.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.818+0000 7f932da00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103f60 0x7f93281a1fa0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f931400e9f0 tx=0x7f931400ed00 comp rx=0 tx=0).stop 2024-04-26T20:10:38.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.819+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 shutdown_connections 2024-04-26T20:10:38.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.819+0000 7f932da00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9328108690 0x7f932819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:38.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.819+0000 7f932da00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f930c0a3d50 0x7f930c0a6210 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:38.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.819+0000 7f932da00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93281035b0 0x7f93281a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:38.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.819+0000 7f932da00700 1 --2- 172.21.15.94:0/505145612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9328103f60 0x7f93281a1fa0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:38.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.819+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 >> 172.21.15.94:0/505145612 conn(0x7f93280feea0 msgr2=0x7f9328101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:38.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.819+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 shutdown_connections 2024-04-26T20:10:38.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:36.819+0000 7f932da00700 1 -- 172.21.15.94:0/505145612 wait complete. 2024-04-26T20:10:38.629 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:37 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/505145612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T20:10:38.637 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:37 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/505145612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T20:10:38.644 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:37 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/505145612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T20:10:38.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.486+0000 7fd685a00700 1 -- 172.21.15.94:0/4011762994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd68010a410 msgr2=0x7fd680116920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:38.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.486+0000 7fd685a00700 1 --2- 172.21.15.94:0/4011762994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd68010a410 0x7fd680116920 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd67800b0a0 tx=0x7fd67800b3b0 comp rx=0 tx=0).stop 2024-04-26T20:10:38.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.486+0000 7fd685a00700 1 -- 172.21.15.94:0/4011762994 shutdown_connections 2024-04-26T20:10:38.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.486+0000 7fd685a00700 1 --2- 172.21.15.94:0/4011762994 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd68010a410 0x7fd680116920 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:38.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.486+0000 7fd685a00700 1 --2- 172.21.15.94:0/4011762994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd68010fa50 0x7fd68010fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.486+0000 7fd685a00700 1 --2- 172.21.15.94:0/4011762994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd680109a50 0x7fd680109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:38.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.486+0000 7fd685a00700 1 -- 172.21.15.94:0/4011762994 >> 172.21.15.94:0/4011762994 conn(0x7fd680076bf0 msgr2=0x7fd680108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:38.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.486+0000 7fd685a00700 1 -- 172.21.15.94:0/4011762994 shutdown_connections 2024-04-26T20:10:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.486+0000 7fd685a00700 1 -- 172.21.15.94:0/4011762994 wait complete. 2024-04-26T20:10:38.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.487+0000 7fd685a00700 1 Processor -- start 2024-04-26T20:10:38.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.487+0000 7fd685a00700 1 -- start start 2024-04-26T20:10:38.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd685a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd680109a50 0x7fd6801a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:38.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd685a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd68010a410 0x7fd6801a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:38.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd685a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd68010fa50 0x7fd6801a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:38.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd685a00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd680119c90 con 0x7fd680109a50 2024-04-26T20:10:38.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd685a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd680119b10 con 0x7fd68010fa50 2024-04-26T20:10:38.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd685a00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd680119e10 con 0x7fd68010a410 2024-04-26T20:10:38.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd67ea00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd680109a50 0x7fd6801a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:38.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd67ea00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd680109a50 0x7fd6801a0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50010/0 (socket says 172.21.15.94:50010) 2024-04-26T20:10:38.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd67e000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd68010a410 0x7fd6801a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:38.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd67ea00700 1 -- 172.21.15.94:0/2584743340 learned_addr learned my addr 172.21.15.94:0/2584743340 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:39.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.488+0000 7fd67f400700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd68010fa50 0x7fd6801a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:39.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd67ea00700 1 -- 172.21.15.94:0/2584743340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd68010a410 msgr2=0x7fd6801a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:39.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd67ea00700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd68010a410 0x7fd6801a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:39.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd67ea00700 1 -- 172.21.15.94:0/2584743340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd68010fa50 msgr2=0x7fd6801a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:39.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd67ea00700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd68010fa50 0x7fd6801a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:39.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd67ea00700 1 -- 172.21.15.94:0/2584743340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6801a5070 con 0x7fd680109a50 2024-04-26T20:10:39.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd67ea00700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd680109a50 0x7fd6801a0060 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd67000e9a0 tx=0x7fd67000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:39.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd677400700 1 -- 172.21.15.94:0/2584743340 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67000ca10 con 0x7fd680109a50 2024-04-26T20:10:39.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd677400700 1 -- 172.21.15.94:0/2584743340 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67000cbb0 con 0x7fd680109a50 2024-04-26T20:10:39.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6801a5360 con 0x7fd680109a50 2024-04-26T20:10:39.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.489+0000 7fd677400700 1 -- 172.21.15.94:0/2584743340 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670018680 con 0x7fd680109a50 2024-04-26T20:10:39.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.490+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6801acbe0 con 0x7fd680109a50 2024-04-26T20:10:39.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.491+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd68006a110 con 0x7fd680109a50 2024-04-26T20:10:39.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.493+0000 7fd677400700 1 -- 172.21.15.94:0/2584743340 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd670010430 con 0x7fd680109a50 2024-04-26T20:10:39.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.495+0000 7fd677400700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd6640a3d50 0x7fd6640a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:39.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.496+0000 7fd677400700 1 -- 172.21.15.94:0/2584743340 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fd6700d0d40 con 0x7fd680109a50 2024-04-26T20:10:39.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.496+0000 7fd67e000700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd6640a3d50 0x7fd6640a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:39.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.496+0000 7fd67e000700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd6640a3d50 0x7fd6640a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd680078af0 tx=0x7fd66c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:39.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.498+0000 7fd677400700 1 -- 172.21.15.94:0/2584743340 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd67009d190 con 0x7fd680109a50 2024-04-26T20:10:39.524 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:38 smithi204 ceph-mon[32618]: pgmap v399: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:10:39.579 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:38 smithi103 ceph-mon[32635]: pgmap v399: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:10:39.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.732+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fd68002e490 con 0x7fd680109a50 2024-04-26T20:10:39.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.733+0000 7fd677400700 1 -- 172.21.15.94:0/2584743340 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd67009ca90 con 0x7fd680109a50 2024-04-26T20:10:39.679 INFO:teuthology.orchestra.run.smithi094.stdout:287762808887 2024-04-26T20:10:39.687 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:38 smithi094 ceph-mon[28050]: pgmap v399: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:10:39.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd6640a3d50 msgr2=0x7fd6640a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:39.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd6640a3d50 0x7fd6640a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd680078af0 tx=0x7fd66c009260 comp rx=0 tx=0).stop 2024-04-26T20:10:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd680109a50 msgr2=0x7fd6801a0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd680109a50 0x7fd6801a0060 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd67000e9a0 tx=0x7fd67000ecb0 comp rx=0 tx=0).stop 2024-04-26T20:10:39.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 shutdown_connections 2024-04-26T20:10:39.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd6640a3d50 0x7fd6640a6210 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:39.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd68010fa50 0x7fd6801a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:39.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd680109a50 0x7fd6801a0060 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:39.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 --2- 172.21.15.94:0/2584743340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd68010a410 0x7fd6801a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:39.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 >> 172.21.15.94:0/2584743340 conn(0x7fd680076bf0 msgr2=0x7fd6800771b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:39.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 shutdown_connections 2024-04-26T20:10:39.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:38.737+0000 7fd685a00700 1 -- 172.21.15.94:0/2584743340 wait complete. 2024-04-26T20:10:39.944 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:39 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2584743340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T20:10:40.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:39 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2584743340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T20:10:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:39 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2584743340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T20:10:40.449 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411419 got 77309411423 for osd.2 2024-04-26T20:10:40.470 DEBUG:teuthology.parallel:result is None 2024-04-26T20:10:40.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.659+0000 7f858f200700 1 -- 172.21.15.94:0/2640914573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f859010b5f0 msgr2=0x7f859010b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:40.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.659+0000 7f858f200700 1 --2- 172.21.15.94:0/2640914573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f859010b5f0 0x7f859010b9d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f85800099c0 tx=0x7f8580009cd0 comp rx=0 tx=0).stop 2024-04-26T20:10:40.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.659+0000 7f858f200700 1 -- 172.21.15.94:0/2640914573 shutdown_connections 2024-04-26T20:10:40.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.659+0000 7f858f200700 1 --2- 172.21.15.94:0/2640914573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8590105f00 0x7f85901123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:40.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.659+0000 7f858f200700 1 --2- 172.21.15.94:0/2640914573 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f859010b5f0 0x7f859010b9d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:40.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.659+0000 7f858f200700 1 --2- 172.21.15.94:0/2640914573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8590105540 0x7f85901059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:40.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.659+0000 7f858f200700 1 -- 172.21.15.94:0/2640914573 >> 172.21.15.94:0/2640914573 conn(0x7f8590101050 msgr2=0x7f8590103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:40.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.659+0000 7f858f200700 1 -- 172.21.15.94:0/2640914573 shutdown_connections 2024-04-26T20:10:40.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.659+0000 7f858f200700 1 -- 172.21.15.94:0/2640914573 wait complete. 2024-04-26T20:10:40.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.661+0000 7f858f200700 1 Processor -- start 2024-04-26T20:10:40.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.661+0000 7f858f200700 1 -- start start 2024-04-26T20:10:40.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.661+0000 7f858f200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8590105540 0x7f85901a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:40.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.661+0000 7f858f200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8590105f00 0x7f85901a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.661+0000 7f858f200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f859010b5f0 0x7f859019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:40.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.661+0000 7f858f200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8590115750 con 0x7f8590105f00 2024-04-26T20:10:40.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.661+0000 7f858f200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85901155d0 con 0x7f8590105540 2024-04-26T20:10:40.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.661+0000 7f858f200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85901158d0 con 0x7f859010b5f0 2024-04-26T20:10:40.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858de00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8590105540 0x7f85901a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:41.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858d400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8590105f00 0x7f85901a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:41.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858de00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8590105540 0x7f85901a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:35302/0 (socket says 172.21.15.94:35302) 2024-04-26T20:10:41.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858de00700 1 -- 172.21.15.94:0/3338064972 learned_addr learned my addr 172.21.15.94:0/3338064972 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:41.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858e800700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f859010b5f0 0x7f859019ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:41.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858de00700 1 -- 172.21.15.94:0/3338064972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f859010b5f0 msgr2=0x7f859019ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:41.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858de00700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f859010b5f0 0x7f859019ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:41.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858de00700 1 -- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8590105f00 msgr2=0x7f85901a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:41.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858de00700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8590105f00 0x7f85901a1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:41.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858de00700 1 -- 172.21.15.94:0/3338064972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859019c170 con 0x7f8590105540 2024-04-26T20:10:41.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858e800700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f859010b5f0 0x7f859019ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T20:10:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858d400700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8590105f00 0x7f85901a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:10:41.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.662+0000 7f858de00700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8590105540 0x7f85901a1850 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8580006010 tx=0x7f85800055f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:41.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.663+0000 7f8586a00700 1 -- 172.21.15.94:0/3338064972 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8580026070 con 0x7f8590105540 2024-04-26T20:10:41.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.663+0000 7f8586a00700 1 -- 172.21.15.94:0/3338064972 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f858000f460 con 0x7f8590105540 2024-04-26T20:10:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.663+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f859019c400 con 0x7f8590105540 2024-04-26T20:10:41.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.663+0000 7f8586a00700 1 -- 172.21.15.94:0/3338064972 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8580009e60 con 0x7f8590105540 2024-04-26T20:10:41.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.663+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85901a8720 con 0x7f8590105540 2024-04-26T20:10:41.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.664+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f859010da80 con 0x7f8590105540 2024-04-26T20:10:41.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.667+0000 7f8586a00700 1 -- 172.21.15.94:0/3338064972 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8580029070 con 0x7f8590105540 2024-04-26T20:10:41.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.668+0000 7f8586a00700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85740a3cd0 0x7f85740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:41.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.668+0000 7f8586a00700 1 -- 172.21.15.94:0/3338064972 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f85800d1600 con 0x7f8590105540 2024-04-26T20:10:41.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.668+0000 7f858d400700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85740a3cd0 0x7f85740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:41.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.669+0000 7f858d400700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85740a3cd0 0x7f85740a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f859010ac70 tx=0x7f857c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:41.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.670+0000 7f8586a00700 1 -- 172.21.15.94:0/3338064972 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8580030020 con 0x7f8590105540 2024-04-26T20:10:41.585 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:40 smithi204 ceph-mon[32618]: pgmap v400: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T20:10:41.593 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:40 smithi103 ceph-mon[32635]: pgmap v400: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T20:10:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.915+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f859019cf40 con 0x7f8590105540 2024-04-26T20:10:41.695 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:40 smithi094 ceph-mon[28050]: pgmap v400: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T20:10:41.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.915+0000 7f8586a00700 1 -- 172.21.15.94:0/3338064972 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8580013070 con 0x7f8590105540 2024-04-26T20:10:41.811 INFO:teuthology.orchestra.run.smithi094.stdout:55834574949 2024-04-26T20:10:41.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85740a3cd0 msgr2=0x7f85740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85740a3cd0 0x7f85740a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f859010ac70 tx=0x7f857c0073d0 comp rx=0 tx=0).stop 2024-04-26T20:10:41.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8590105540 msgr2=0x7f85901a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:41.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8590105540 0x7f85901a1850 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8580006010 tx=0x7f85800055f0 comp rx=0 tx=0).stop 2024-04-26T20:10:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 shutdown_connections 2024-04-26T20:10:41.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f859010b5f0 0x7f859019ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:41.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f85740a3cd0 0x7f85740a6190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:41.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8590105540 0x7f85901a1850 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:41.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 --2- 172.21.15.94:0/3338064972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8590105f00 0x7f85901a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:41.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 >> 172.21.15.94:0/3338064972 conn(0x7f8590101050 msgr2=0x7f85901027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:41.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 shutdown_connections 2024-04-26T20:10:41.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:40.919+0000 7f858f200700 1 -- 172.21.15.94:0/3338064972 wait complete. 2024-04-26T20:10:41.959 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808883 got 287762808887 for osd.9 2024-04-26T20:10:42.033 DEBUG:teuthology.parallel:result is None 2024-04-26T20:10:42.119 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:41 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3338064972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T20:10:42.179 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:41 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3338064972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T20:10:42.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:41 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3338064972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T20:10:43.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:42 smithi103 ceph-mon[32635]: pgmap v401: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:10:43.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:42 smithi204 ceph-mon[32618]: pgmap v401: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:10:43.172 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:42 smithi094 ceph-mon[28050]: pgmap v401: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:10:43.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.337+0000 7fd97c200700 1 -- 172.21.15.94:0/2489960748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 msgr2=0x7fd974105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:43.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.337+0000 7fd97c200700 1 --2- 172.21.15.94:0/2489960748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 0x7fd974105ac0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd960009990 tx=0x7fd960009ca0 comp rx=0 tx=0).stop 2024-04-26T20:10:43.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.337+0000 7fd97c200700 1 -- 172.21.15.94:0/2489960748 shutdown_connections 2024-04-26T20:10:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.337+0000 7fd97c200700 1 --2- 172.21.15.94:0/2489960748 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd974106000 0x7fd974112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:43.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.337+0000 7fd97c200700 1 --2- 172.21.15.94:0/2489960748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd97410b640 0x7fd97410ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:43.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.337+0000 7fd97c200700 1 --2- 172.21.15.94:0/2489960748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 0x7fd974105ac0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:43.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.337+0000 7fd97c200700 1 -- 172.21.15.94:0/2489960748 >> 172.21.15.94:0/2489960748 conn(0x7fd974101130 msgr2=0x7fd974103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:43.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.337+0000 7fd97c200700 1 -- 172.21.15.94:0/2489960748 shutdown_connections 2024-04-26T20:10:43.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.337+0000 7fd97c200700 1 -- 172.21.15.94:0/2489960748 wait complete. 2024-04-26T20:10:43.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.339+0000 7fd97c200700 1 Processor -- start 2024-04-26T20:10:43.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.339+0000 7fd97c200700 1 -- start start 2024-04-26T20:10:43.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.339+0000 7fd97c200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 0x7fd97419bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:43.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.339+0000 7fd97c200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd974106000 0x7fd97419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:43.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.339+0000 7fd97c200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd97410b640 0x7fd9741a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:43.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.339+0000 7fd97c200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd9741158d0 con 0x7fd97410b640 2024-04-26T20:10:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.339+0000 7fd97c200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd974115750 con 0x7fd974105640 2024-04-26T20:10:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.339+0000 7fd97c200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd974115a50 con 0x7fd974106000 2024-04-26T20:10:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd979800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 0x7fd97419bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:43.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd979800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 0x7fd97419bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:54028/0 (socket says 172.21.15.94:54028) 2024-04-26T20:10:43.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd979800700 1 -- 172.21.15.94:0/1432482038 learned_addr learned my addr 172.21.15.94:0/1432482038 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:43.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd978e00700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd974106000 0x7fd97419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:43.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd97a200700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd97410b640 0x7fd9741a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:43.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd979800700 1 -- 172.21.15.94:0/1432482038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd974106000 msgr2=0x7fd97419c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:43.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd979800700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd974106000 0x7fd97419c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:43.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd979800700 1 -- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd97410b640 msgr2=0x7fd9741a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:43.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd979800700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd97410b640 0x7fd9741a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:43.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd979800700 1 -- 172.21.15.94:0/1432482038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9741a0d50 con 0x7fd974105640 2024-04-26T20:10:43.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.340+0000 7fd979800700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 0x7fd97419bd40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd96800e9c0 tx=0x7fd96800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:43.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.341+0000 7fd972000700 1 -- 172.21.15.94:0/1432482038 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96800cbd0 con 0x7fd974105640 2024-04-26T20:10:44.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.341+0000 7fd972000700 1 -- 172.21.15.94:0/1432482038 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd96800ee60 con 0x7fd974105640 2024-04-26T20:10:44.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.341+0000 7fd972000700 1 -- 172.21.15.94:0/1432482038 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9680055c0 con 0x7fd974105640 2024-04-26T20:10:44.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.341+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9741a1040 con 0x7fd974105640 2024-04-26T20:10:44.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.341+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9741a88c0 con 0x7fd974105640 2024-04-26T20:10:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.342+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd97410dbd0 con 0x7fd974105640 2024-04-26T20:10:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.344+0000 7fd972000700 1 -- 172.21.15.94:0/1432482038 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd968004750 con 0x7fd974105640 2024-04-26T20:10:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.346+0000 7fd972000700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd95c0a3cd0 0x7fd95c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:44.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.346+0000 7fd972000700 1 -- 172.21.15.94:0/1432482038 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fd9680d0de0 con 0x7fd974105640 2024-04-26T20:10:44.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.346+0000 7fd978e00700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd95c0a3cd0 0x7fd95c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:44.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.347+0000 7fd978e00700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd95c0a3cd0 0x7fd95c0a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd974103770 tx=0x7fd96000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:44.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.348+0000 7fd972000700 1 -- 172.21.15.94:0/1432482038 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd968054060 con 0x7fd974105640 2024-04-26T20:10:44.222 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:43 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1432482038' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T20:10:44.230 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:43 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1432482038' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T20:10:44.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.591+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd97410d5b0 con 0x7fd974105640 2024-04-26T20:10:44.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.593+0000 7fd972000700 1 -- 172.21.15.94:0/1432482038 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd96809d230 con 0x7fd974105640 2024-04-26T20:10:44.331 INFO:teuthology.orchestra.run.smithi094.stdout:34359738473 2024-04-26T20:10:44.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.596+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd95c0a3cd0 msgr2=0x7fd95c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:44.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.596+0000 7fd97c200700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd95c0a3cd0 0x7fd95c0a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd974103770 tx=0x7fd96000b260 comp rx=0 tx=0).stop 2024-04-26T20:10:44.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.596+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 msgr2=0x7fd97419bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:44.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.596+0000 7fd97c200700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 0x7fd97419bd40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd96800e9c0 tx=0x7fd96800ecd0 comp rx=0 tx=0).stop 2024-04-26T20:10:44.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.597+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 shutdown_connections 2024-04-26T20:10:44.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.597+0000 7fd97c200700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd97410b640 0x7fd9741a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:44.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.597+0000 7fd97c200700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd95c0a3cd0 0x7fd95c0a6190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:44.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.597+0000 7fd97c200700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd974105640 0x7fd97419bd40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:44.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.597+0000 7fd97c200700 1 --2- 172.21.15.94:0/1432482038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd974106000 0x7fd97419c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:44.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.597+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 >> 172.21.15.94:0/1432482038 conn(0x7fd974101130 msgr2=0x7fd9741018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:44.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.597+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 shutdown_connections 2024-04-26T20:10:44.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.597+0000 7fd97c200700 1 -- 172.21.15.94:0/1432482038 wait complete. 2024-04-26T20:10:44.631 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574945 got 55834574949 for osd.1 2024-04-26T20:10:44.638 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:43 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1432482038' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T20:10:44.645 DEBUG:teuthology.parallel:result is None 2024-04-26T20:10:44.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.965+0000 7f3cdae00700 1 -- 172.21.15.94:0/2089111893 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd4101800 msgr2=0x7f3cd4101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:44.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.965+0000 7f3cdae00700 1 --2- 172.21.15.94:0/2089111893 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd4101800 0x7f3cd4101be0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40099c0 tx=0x7f3cc4009cd0 comp rx=0 tx=0).stop 2024-04-26T20:10:44.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.965+0000 7f3cdae00700 1 -- 172.21.15.94:0/2089111893 shutdown_connections 2024-04-26T20:10:44.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.965+0000 7f3cdae00700 1 --2- 172.21.15.94:0/2089111893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd41098e0 0x7f3cd4110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:44.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.965+0000 7f3cdae00700 1 --2- 172.21.15.94:0/2089111893 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd4101800 0x7f3cd4101be0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:44.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.965+0000 7f3cdae00700 1 --2- 172.21.15.94:0/2089111893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cd41021b0 0x7f3cd4109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:44.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.965+0000 7f3cdae00700 1 -- 172.21.15.94:0/2089111893 >> 172.21.15.94:0/2089111893 conn(0x7f3cd4078520 msgr2=0x7f3cd40ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:44.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.965+0000 7f3cdae00700 1 -- 172.21.15.94:0/2089111893 shutdown_connections 2024-04-26T20:10:44.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.966+0000 7f3cdae00700 1 -- 172.21.15.94:0/2089111893 wait complete. 2024-04-26T20:10:44.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.966+0000 7f3cdae00700 1 Processor -- start 2024-04-26T20:10:44.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cdae00700 1 -- start start 2024-04-26T20:10:44.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cdae00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cd4101800 0x7f3cd419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:44.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cdae00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd41021b0 0x7f3cd419fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:44.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cdae00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd41098e0 0x7f3cd4199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cdae00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd4113620 con 0x7f3cd41021b0 2024-04-26T20:10:44.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cdae00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd41134a0 con 0x7f3cd41098e0 2024-04-26T20:10:44.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cdae00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd41137a0 con 0x7f3cd4101800 2024-04-26T20:10:44.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cd3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd41021b0 0x7f3cd419fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:45.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cd3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd41021b0 0x7f3cd419fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45194/0 (socket says 172.21.15.94:45194) 2024-04-26T20:10:45.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cd3400700 1 -- 172.21.15.94:0/1074880291 learned_addr learned my addr 172.21.15.94:0/1074880291 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:45.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cd8e00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd41098e0 0x7f3cd4199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:45.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.967+0000 7f3cd3e00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cd4101800 0x7f3cd419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:45.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd3400700 1 -- 172.21.15.94:0/1074880291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cd4101800 msgr2=0x7f3cd419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:45.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd3400700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cd4101800 0x7f3cd419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:45.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd3400700 1 -- 172.21.15.94:0/1074880291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd41098e0 msgr2=0x7f3cd4199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:45.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd3400700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd41098e0 0x7f3cd4199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:45.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd3400700 1 -- 172.21.15.94:0/1074880291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd419a0a0 con 0x7f3cd41021b0 2024-04-26T20:10:45.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd3e00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cd4101800 0x7f3cd419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:10:45.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd8e00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd41098e0 0x7f3cd4199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:10:45.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd3400700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd41021b0 0x7f3cd419fcc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000d980 tx=0x7f3cc000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:45.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd0c00700 1 -- 172.21.15.94:0/1074880291 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0017070 con 0x7f3cd41021b0 2024-04-26T20:10:45.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd0c00700 1 -- 172.21.15.94:0/1074880291 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cc0009de0 con 0x7f3cd41021b0 2024-04-26T20:10:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd419a390 con 0x7f3cd41021b0 2024-04-26T20:10:45.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.968+0000 7f3cd0c00700 1 -- 172.21.15.94:0/1074880291 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc000f3c0 con 0x7f3cd41021b0 2024-04-26T20:10:45.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.969+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd41a6650 con 0x7f3cd41021b0 2024-04-26T20:10:45.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.970+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd4102930 con 0x7f3cd41021b0 2024-04-26T20:10:45.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.972+0000 7f3cd0c00700 1 -- 172.21.15.94:0/1074880291 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3cc0009930 con 0x7f3cd41021b0 2024-04-26T20:10:45.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.974+0000 7f3cd0c00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3cb80a3cd0 0x7f3cb80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:45.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.974+0000 7f3cd0c00700 1 -- 172.21.15.94:0/1074880291 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f3cc00d0440 con 0x7f3cd41021b0 2024-04-26T20:10:45.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.974+0000 7f3cd3e00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3cb80a3cd0 0x7f3cb80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:45.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.975+0000 7f3cd3e00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3cb80a3cd0 0x7f3cb80a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4000c00 tx=0x7f3cc4005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:45.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:43.979+0000 7f3cd0c00700 1 -- 172.21.15.94:0/1074880291 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3cc009c8f0 con 0x7f3cd41021b0 2024-04-26T20:10:45.471 INFO:teuthology.orchestra.run.smithi094.stdout:219043332164 2024-04-26T20:10:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.214+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f3cd402e490 con 0x7f3cd41021b0 2024-04-26T20:10:45.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.215+0000 7f3cd0c00700 1 -- 172.21.15.94:0/1074880291 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3cc009c1f0 con 0x7f3cd41021b0 2024-04-26T20:10:45.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3cb80a3cd0 msgr2=0x7f3cb80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:45.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3cb80a3cd0 0x7f3cb80a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4000c00 tx=0x7f3cc4005ab0 comp rx=0 tx=0).stop 2024-04-26T20:10:45.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd41021b0 msgr2=0x7f3cd419fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:45.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd41021b0 0x7f3cd419fcc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000d980 tx=0x7f3cc000dc90 comp rx=0 tx=0).stop 2024-04-26T20:10:45.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 shutdown_connections 2024-04-26T20:10:45.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3cb80a3cd0 0x7f3cb80a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:45.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd41098e0 0x7f3cd4199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cd4101800 0x7f3cd419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:45.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 --2- 172.21.15.94:0/1074880291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd41021b0 0x7f3cd419fcc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:45.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 >> 172.21.15.94:0/1074880291 conn(0x7f3cd4078520 msgr2=0x7f3cd4107510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:45.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.218+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 shutdown_connections 2024-04-26T20:10:45.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:44.219+0000 7f3cdae00700 1 -- 172.21.15.94:0/1074880291 wait complete. 2024-04-26T20:10:45.842 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:44 smithi094 ceph-mon[28050]: pgmap v402: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:45.851 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:44 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1074880291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T20:10:45.864 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:44 smithi204 ceph-mon[32618]: pgmap v402: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:45.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:44 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1074880291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T20:10:45.944 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:44 smithi103 ceph-mon[32635]: pgmap v402: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:45.950 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:44 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1074880291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T20:10:45.958 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:45 smithi094 ceph-mon[28050]: pgmap v403: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:46.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:45 smithi103 ceph-mon[32635]: pgmap v403: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:46.112 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738467 got 34359738473 for osd.0 2024-04-26T20:10:46.142 DEBUG:teuthology.parallel:result is None 2024-04-26T20:10:46.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:45 smithi204 ceph-mon[32618]: pgmap v403: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:46.448 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:46.464 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:46.518 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:46.580 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:46.585 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:10:46.698 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332158 got 219043332164 for osd.7 2024-04-26T20:10:46.705 DEBUG:teuthology.parallel:result is None 2024-04-26T20:10:47.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.331+0000 7f7597000700 1 -- 172.21.15.94:0/1992098781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75901094c0 msgr2=0x7f75901098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:47.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.331+0000 7f7597000700 1 --2- 172.21.15.94:0/1992098781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75901094c0 0x7f75901098a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f75800098f0 tx=0x7f7580009c00 comp rx=0 tx=0).stop 2024-04-26T20:10:47.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.331+0000 7f7597000700 1 -- 172.21.15.94:0/1992098781 shutdown_connections 2024-04-26T20:10:47.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.331+0000 7f7597000700 1 --2- 172.21.15.94:0/1992098781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7590103e30 0x7f7590110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.331+0000 7f7597000700 1 --2- 172.21.15.94:0/1992098781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75901094c0 0x7f75901098a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:47.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.331+0000 7f7597000700 1 --2- 172.21.15.94:0/1992098781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7590103470 0x7f75901038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:47.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.331+0000 7f7597000700 1 -- 172.21.15.94:0/1992098781 >> 172.21.15.94:0/1992098781 conn(0x7f759007a580 msgr2=0x7f759007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:47.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.331+0000 7f7597000700 1 -- 172.21.15.94:0/1992098781 shutdown_connections 2024-04-26T20:10:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.331+0000 7f7597000700 1 -- 172.21.15.94:0/1992098781 wait complete. 2024-04-26T20:10:47.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.332+0000 7f7597000700 1 Processor -- start 2024-04-26T20:10:47.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.332+0000 7f7597000700 1 -- start start 2024-04-26T20:10:47.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f7597000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7590103470 0x7f759019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:47.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f7597000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7590103e30 0x7f759019c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f7597000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75901094c0 0x7f75901a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:47.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f7597000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7590113610 con 0x7f75901094c0 2024-04-26T20:10:47.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f7597000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7590113490 con 0x7f7590103e30 2024-04-26T20:10:47.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f7597000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7590113790 con 0x7f7590103470 2024-04-26T20:10:47.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f758fe00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7590103470 0x7f759019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:47.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f758f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7590103e30 0x7f759019c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:47.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f758f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7590103e30 0x7f759019c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:54062/0 (socket says 172.21.15.94:54062) 2024-04-26T20:10:47.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f7595000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75901094c0 0x7f75901a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:47.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.333+0000 7f758f400700 1 -- 172.21.15.94:0/3614995432 learned_addr learned my addr 172.21.15.94:0/3614995432 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:47.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.334+0000 7f7595000700 1 -- 172.21.15.94:0/3614995432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7590103470 msgr2=0x7f759019be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:47.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.334+0000 7f7595000700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7590103470 0x7f759019be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:47.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.334+0000 7f7595000700 1 -- 172.21.15.94:0/3614995432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7590103e30 msgr2=0x7f759019c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:47.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.334+0000 7f7595000700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7590103e30 0x7f759019c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:47.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.334+0000 7f7595000700 1 -- 172.21.15.94:0/3614995432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75901a0e70 con 0x7f75901094c0 2024-04-26T20:10:47.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.334+0000 7f758f400700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7590103e30 0x7f759019c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:10:48.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.334+0000 7f7595000700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75901094c0 0x7f75901a0730 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7588009cd0 tx=0x7f758800c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:48.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.335+0000 7f758cc00700 1 -- 172.21.15.94:0/3614995432 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758800cd10 con 0x7f75901094c0 2024-04-26T20:10:48.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.335+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75901a1100 con 0x7f75901094c0 2024-04-26T20:10:48.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.335+0000 7f758cc00700 1 -- 172.21.15.94:0/3614995432 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7588003a00 con 0x7f75901094c0 2024-04-26T20:10:48.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.335+0000 7f758cc00700 1 -- 172.21.15.94:0/3614995432 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588004520 con 0x7f75901094c0 2024-04-26T20:10:48.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.335+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75901a89e0 con 0x7f75901094c0 2024-04-26T20:10:48.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.336+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f759006a110 con 0x7f75901094c0 2024-04-26T20:10:48.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.339+0000 7f758cc00700 1 -- 172.21.15.94:0/3614995432 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f758801a020 con 0x7f75901094c0 2024-04-26T20:10:48.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.340+0000 7f758cc00700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75740a3d50 0x7f75740a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:48.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.340+0000 7f758cc00700 1 -- 172.21.15.94:0/3614995432 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f75880d0c20 con 0x7f75901094c0 2024-04-26T20:10:48.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.341+0000 7f758fe00700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75740a3d50 0x7f75740a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.341+0000 7f758fe00700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75740a3d50 0x7f75740a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f75800098c0 tx=0x7f7580019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:48.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.343+0000 7f758cc00700 1 -- 172.21.15.94:0/3614995432 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7588010040 con 0x7f75901094c0 2024-04-26T20:10:48.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:47 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3614995432' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T20:10:48.237 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:47 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3614995432' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T20:10:48.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.583+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f759002e490 con 0x7f75901094c0 2024-04-26T20:10:48.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.584+0000 7f758cc00700 1 -- 172.21.15.94:0/3614995432 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f758809d070 con 0x7f75901094c0 2024-04-26T20:10:48.335 INFO:teuthology.orchestra.run.smithi094.stdout:343597383727 2024-04-26T20:10:48.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.586+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75740a3d50 msgr2=0x7f75740a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:48.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.586+0000 7f7597000700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75740a3d50 0x7f75740a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f75800098c0 tx=0x7f7580019040 comp rx=0 tx=0).stop 2024-04-26T20:10:48.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.586+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75901094c0 msgr2=0x7f75901a0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:48.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.586+0000 7f7597000700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75901094c0 0x7f75901a0730 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7588009cd0 tx=0x7f758800c620 comp rx=0 tx=0).stop 2024-04-26T20:10:48.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.587+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 shutdown_connections 2024-04-26T20:10:48.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.587+0000 7f7597000700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75901094c0 0x7f75901a0730 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:48.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.587+0000 7f7597000700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7590103470 0x7f759019be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:48.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.587+0000 7f7597000700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f75740a3d50 0x7f75740a6210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:48.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.587+0000 7f7597000700 1 --2- 172.21.15.94:0/3614995432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7590103e30 0x7f759019c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:48.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.587+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 >> 172.21.15.94:0/3614995432 conn(0x7f759007a580 msgr2=0x7f759010e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.588+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 shutdown_connections 2024-04-26T20:10:48.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:47.588+0000 7f7597000700 1 -- 172.21.15.94:0/3614995432 wait complete. 2024-04-26T20:10:48.584 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:47 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3614995432' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T20:10:48.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:48 smithi094 ceph-mon[28050]: pgmap v404: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T20:10:49.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:48 smithi103 ceph-mon[32635]: pgmap v404: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T20:10:49.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:48 smithi204 ceph-mon[32618]: pgmap v404: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T20:10:50.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:50 smithi094 ceph-mon[28050]: pgmap v405: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:51.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:50 smithi103 ceph-mon[32635]: pgmap v405: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:51.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:50 smithi204 ceph-mon[32618]: pgmap v405: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:51.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.547+0000 7f17bb200700 1 -- 172.21.15.94:0/3241702504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc10a320 msgr2=0x7f17bc1167c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:51.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.547+0000 7f17bb200700 1 --2- 172.21.15.94:0/3241702504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc10a320 0x7f17bc1167c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f17b400b0a0 tx=0x7f17b400b3b0 comp rx=0 tx=0).stop 2024-04-26T20:10:51.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.547+0000 7f17bb200700 1 -- 172.21.15.94:0/3241702504 shutdown_connections 2024-04-26T20:10:51.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.547+0000 7f17bb200700 1 --2- 172.21.15.94:0/3241702504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc10a320 0x7f17bc1167c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:51.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.547+0000 7f17bb200700 1 --2- 172.21.15.94:0/3241702504 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17bc10e830 0x7f17bc10ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:51.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.547+0000 7f17bb200700 1 --2- 172.21.15.94:0/3241702504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17bc109960 0x7f17bc109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:51.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.548+0000 7f17bb200700 1 -- 172.21.15.94:0/3241702504 >> 172.21.15.94:0/3241702504 conn(0x7f17bc0791b0 msgr2=0x7f17bc108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:52.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.548+0000 7f17bb200700 1 -- 172.21.15.94:0/3241702504 shutdown_connections 2024-04-26T20:10:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.548+0000 7f17bb200700 1 -- 172.21.15.94:0/3241702504 wait complete. 2024-04-26T20:10:52.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.549+0000 7f17bb200700 1 Processor -- start 2024-04-26T20:10:52.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.549+0000 7f17bb200700 1 -- start start 2024-04-26T20:10:52.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.549+0000 7f17bb200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17bc109960 0x7f17bc1a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:52.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.549+0000 7f17bb200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17bc10a320 0x7f17bc1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:52.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.549+0000 7f17bb200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc10e830 0x7f17bc1a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:52.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.549+0000 7f17bb200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc119b70 con 0x7f17bc10e830 2024-04-26T20:10:52.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.549+0000 7f17bb200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc1199f0 con 0x7f17bc109960 2024-04-26T20:10:52.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.549+0000 7f17bb200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc119cf0 con 0x7f17bc10a320 2024-04-26T20:10:52.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17bc109960 0x7f17bc1a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:52.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17bc109960 0x7f17bc1a0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:59030/0 (socket says 172.21.15.94:59030) 2024-04-26T20:10:52.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17ba800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc10e830 0x7f17bc1a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:52.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9e00700 1 -- 172.21.15.94:0/1516399918 learned_addr learned my addr 172.21.15.94:0/1516399918 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:52.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9400700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17bc10a320 0x7f17bc1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:52.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9e00700 1 -- 172.21.15.94:0/1516399918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17bc10a320 msgr2=0x7f17bc1a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:52.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9e00700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17bc10a320 0x7f17bc1a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:52.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9e00700 1 -- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc10e830 msgr2=0x7f17bc1a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:52.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9e00700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc10e830 0x7f17bc1a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:52.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9e00700 1 -- 172.21.15.94:0/1516399918 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17bc1a5070 con 0x7f17bc109960 2024-04-26T20:10:52.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17ba800700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc10e830 0x7f17bc1a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:10:52.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.550+0000 7f17b9e00700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17bc109960 0x7f17bc1a0060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00e9c0 tx=0x7f17ac00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:52.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.551+0000 7f17b2a00700 1 -- 172.21.15.94:0/1516399918 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac00cbd0 con 0x7f17bc109960 2024-04-26T20:10:52.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.551+0000 7f17b2a00700 1 -- 172.21.15.94:0/1516399918 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17ac00ee60 con 0x7f17bc109960 2024-04-26T20:10:52.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.551+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17bc1a5360 con 0x7f17bc109960 2024-04-26T20:10:52.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.551+0000 7f17b2a00700 1 -- 172.21.15.94:0/1516399918 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac0055c0 con 0x7f17bc109960 2024-04-26T20:10:52.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.551+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17bc1acbe0 con 0x7f17bc109960 2024-04-26T20:10:52.882 INFO:teuthology.orchestra.run.smithi094.stdout:107374182492 2024-04-26T20:10:52.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.552+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17bc06a110 con 0x7f17bc109960 2024-04-26T20:10:52.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.554+0000 7f17b2a00700 1 -- 172.21.15.94:0/1516399918 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f17ac004750 con 0x7f17bc109960 2024-04-26T20:10:52.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.555+0000 7f17b2a00700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f17a00a3cd0 0x7f17a00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:52.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.555+0000 7f17b2a00700 1 -- 172.21.15.94:0/1516399918 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f17ac0d0ba0 con 0x7f17bc109960 2024-04-26T20:10:52.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.556+0000 7f17b9400700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f17a00a3cd0 0x7f17a00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:52.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.556+0000 7f17b9400700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f17a00a3cd0 0x7f17a00a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009680 tx=0x7f17a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:52.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.560+0000 7f17b2a00700 1 -- 172.21.15.94:0/1516399918 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f17ac09cff0 con 0x7f17bc109960 2024-04-26T20:10:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.817+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f17bc1a13c0 con 0x7f17bc109960 2024-04-26T20:10:52.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.818+0000 7f17b2a00700 1 -- 172.21.15.94:0/1516399918 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f17ac09c8f0 con 0x7f17bc109960 2024-04-26T20:10:52.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f17a00a3cd0 msgr2=0x7f17a00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:52.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f17a00a3cd0 0x7f17a00a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009680 tx=0x7f17a8009260 comp rx=0 tx=0).stop 2024-04-26T20:10:52.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17bc109960 msgr2=0x7f17bc1a0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:53.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17bc109960 0x7f17bc1a0060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00e9c0 tx=0x7f17ac00ecd0 comp rx=0 tx=0).stop 2024-04-26T20:10:53.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 shutdown_connections 2024-04-26T20:10:53.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc10e830 0x7f17bc1a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:53.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f17a00a3cd0 0x7f17a00a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:53.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17bc109960 0x7f17bc1a0060 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:53.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 --2- 172.21.15.94:0/1516399918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17bc10a320 0x7f17bc1a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:53.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 >> 172.21.15.94:0/1516399918 conn(0x7f17bc0791b0 msgr2=0x7f17bc079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:53.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.821+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 shutdown_connections 2024-04-26T20:10:53.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:51.822+0000 7f17bb200700 1 -- 172.21.15.94:0/1516399918 wait complete. 2024-04-26T20:10:53.218 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:52 smithi094 ceph-mon[28050]: pgmap v406: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:53.225 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:53.234 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:52 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:53.247 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:52 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1516399918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T20:10:53.256 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:52 smithi204 ceph-mon[32618]: pgmap v406: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:53.267 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:53.276 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:52 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:53.285 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:52 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1516399918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T20:10:53.339 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:52 smithi103 ceph-mon[32635]: pgmap v406: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:53.376 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:53.440 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:52 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:53.448 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:52 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1516399918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T20:10:54.773 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182486 got 107374182492 for osd.3 2024-04-26T20:10:54.863 DEBUG:teuthology.parallel:result is None 2024-04-26T20:10:54.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:54 smithi094 ceph-mon[28050]: pgmap v407: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:54.995 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:10:55.002 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:10:55.010 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:55.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:54 smithi103 ceph-mon[32635]: pgmap v407: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:55.042 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:10:55.048 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:10:55.055 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:55.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:54 smithi204 ceph-mon[32618]: pgmap v407: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T20:10:55.141 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:10:55.152 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:10:55.263 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:10:56.152 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383720 got 343597383727 for osd.11 2024-04-26T20:10:56.234 DEBUG:teuthology.parallel:result is None 2024-04-26T20:10:56.950 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:56 smithi094 ceph-mon[28050]: pgmap v408: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:10:57.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:56 smithi103 ceph-mon[32635]: pgmap v408: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:10:57.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:56 smithi204 ceph-mon[32618]: pgmap v408: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:10:57.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.498+0000 7f78a6e00700 1 -- 172.21.15.94:0/3704593097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a006bad0 msgr2=0x7f78a01102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:57.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.498+0000 7f78a6e00700 1 --2- 172.21.15.94:0/3704593097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a006bad0 0x7f78a01102e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7898009f90 tx=0x7f789800b330 comp rx=0 tx=0).stop 2024-04-26T20:10:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.498+0000 7f78a6e00700 1 -- 172.21.15.94:0/3704593097 shutdown_connections 2024-04-26T20:10:57.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.498+0000 7f78a6e00700 1 --2- 172.21.15.94:0/3704593097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a006bad0 0x7f78a01102e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:57.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.498+0000 7f78a6e00700 1 --2- 172.21.15.94:0/3704593097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78a01094a0 0x7f78a0109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:57.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.498+0000 7f78a6e00700 1 --2- 172.21.15.94:0/3704593097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a006b130 0x7f78a006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:57.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.498+0000 7f78a6e00700 1 -- 172.21.15.94:0/3704593097 >> 172.21.15.94:0/3704593097 conn(0x7f78a00792d0 msgr2=0x7f78a00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:57.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.499+0000 7f78a6e00700 1 -- 172.21.15.94:0/3704593097 shutdown_connections 2024-04-26T20:10:57.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.499+0000 7f78a6e00700 1 -- 172.21.15.94:0/3704593097 wait complete. 2024-04-26T20:10:57.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.500+0000 7f78a6e00700 1 Processor -- start 2024-04-26T20:10:57.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.500+0000 7f78a6e00700 1 -- start start 2024-04-26T20:10:57.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.500+0000 7f78a6e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a006b130 0x7f78a019bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:57.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.500+0000 7f78a6e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78a006bad0 0x7f78a019c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.500+0000 7f78a6e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a01094a0 0x7f78a01a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.500+0000 7f78a6e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f78a01135f0 con 0x7f78a006b130 2024-04-26T20:10:58.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.500+0000 7f78a6e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f78a0113470 con 0x7f78a006bad0 2024-04-26T20:10:58.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.500+0000 7f78a6e00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78a0113770 con 0x7f78a01094a0 2024-04-26T20:10:58.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.500+0000 7f789f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78a006bad0 0x7f78a019c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:58.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f789f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78a006bad0 0x7f78a019c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:59038/0 (socket says 172.21.15.94:59038) 2024-04-26T20:10:58.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f789fe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a006b130 0x7f78a019bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:58.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f789f400700 1 -- 172.21.15.94:0/1225130471 learned_addr learned my addr 172.21.15.94:0/1225130471 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:58.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f78a4e00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a01094a0 0x7f78a01a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:58.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f789f400700 1 -- 172.21.15.94:0/1225130471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a01094a0 msgr2=0x7f78a01a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:58.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f789f400700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a01094a0 0x7f78a01a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:58.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f789f400700 1 -- 172.21.15.94:0/1225130471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a006b130 msgr2=0x7f78a019bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:58.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f789f400700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a006b130 0x7f78a019bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:58.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f789f400700 1 -- 172.21.15.94:0/1225130471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a01a0d30 con 0x7f78a006bad0 2024-04-26T20:10:58.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.501+0000 7f789f400700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78a006bad0 0x7f78a019c260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f788c00eb30 tx=0x7f788c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:58.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.502+0000 7f789cc00700 1 -- 172.21.15.94:0/1225130471 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c00cbd0 con 0x7f78a006bad0 2024-04-26T20:10:58.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.502+0000 7f789cc00700 1 -- 172.21.15.94:0/1225130471 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f788c0044d0 con 0x7f78a006bad0 2024-04-26T20:10:58.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.502+0000 7f789cc00700 1 -- 172.21.15.94:0/1225130471 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c003df0 con 0x7f78a006bad0 2024-04-26T20:10:58.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.502+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a01a1020 con 0x7f78a006bad0 2024-04-26T20:10:58.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.502+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a01a88a0 con 0x7f78a006bad0 2024-04-26T20:10:58.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.503+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a006a110 con 0x7f78a006bad0 2024-04-26T20:10:58.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.506+0000 7f789cc00700 1 -- 172.21.15.94:0/1225130471 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f788c003f90 con 0x7f78a006bad0 2024-04-26T20:10:58.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.507+0000 7f789cc00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78840a3d50 0x7f78840a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:58.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.507+0000 7f789cc00700 1 -- 172.21.15.94:0/1225130471 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f788c0d0d00 con 0x7f78a006bad0 2024-04-26T20:10:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.508+0000 7f789fe00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78840a3d50 0x7f78840a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:58.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.508+0000 7f789fe00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78840a3d50 0x7f78840a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7890005fd0 tx=0x7f7890009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:58.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.509+0000 7f789cc00700 1 -- 172.21.15.94:0/1225130471 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f788c09d150 con 0x7f78a006bad0 2024-04-26T20:10:58.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.757+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f78a002e490 con 0x7f78a006bad0 2024-04-26T20:10:58.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.758+0000 7f789cc00700 1 -- 172.21.15.94:0/1225130471 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f788c09ca50 con 0x7f78a006bad0 2024-04-26T20:10:58.631 INFO:teuthology.orchestra.run.smithi094.stdout:309237645364 2024-04-26T20:10:58.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78840a3d50 msgr2=0x7f78840a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:58.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78840a3d50 0x7f78840a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7890005fd0 tx=0x7f7890009310 comp rx=0 tx=0).stop 2024-04-26T20:10:58.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78a006bad0 msgr2=0x7f78a019c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:58.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78a006bad0 0x7f78a019c260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f788c00eb30 tx=0x7f788c00ee40 comp rx=0 tx=0).stop 2024-04-26T20:10:58.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 shutdown_connections 2024-04-26T20:10:58.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a01094a0 0x7f78a01a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:58.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a006b130 0x7f78a019bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:58.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78840a3d50 0x7f78840a6210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:58.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 --2- 172.21.15.94:0/1225130471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78a006bad0 0x7f78a019c260 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:58.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.761+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 >> 172.21.15.94:0/1225130471 conn(0x7f78a00792d0 msgr2=0x7f78a010e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:58.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.762+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 shutdown_connections 2024-04-26T20:10:58.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.762+0000 7f78a6e00700 1 -- 172.21.15.94:0/1225130471 wait complete. 2024-04-26T20:10:58.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.944+0000 7fdd38c00700 1 -- 172.21.15.94:0/3908752380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 msgr2=0x7fdd34110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.944+0000 7fdd38c00700 1 --2- 172.21.15.94:0/3908752380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 0x7fdd34110250 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00b0d0 tx=0x7fdd2c00b3e0 comp rx=0 tx=0).stop 2024-04-26T20:10:58.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.944+0000 7fdd38c00700 1 -- 172.21.15.94:0/3908752380 shutdown_connections 2024-04-26T20:10:58.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.944+0000 7fdd38c00700 1 --2- 172.21.15.94:0/3908752380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 0x7fdd34110250 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:58.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.944+0000 7fdd38c00700 1 --2- 172.21.15.94:0/3908752380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd341035a0 0x7fdd34103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:58.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.944+0000 7fdd38c00700 1 --2- 172.21.15.94:0/3908752380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd34103f50 0x7fdd34107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:59.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.944+0000 7fdd38c00700 1 -- 172.21.15.94:0/3908752380 >> 172.21.15.94:0/3908752380 conn(0x7fdd340fee70 msgr2=0x7fdd34101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:10:59.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.944+0000 7fdd38c00700 1 -- 172.21.15.94:0/3908752380 shutdown_connections 2024-04-26T20:10:59.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.944+0000 7fdd38c00700 1 -- 172.21.15.94:0/3908752380 wait complete. 2024-04-26T20:10:59.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.945+0000 7fdd38c00700 1 Processor -- start 2024-04-26T20:10:59.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.945+0000 7fdd38c00700 1 -- start start 2024-04-26T20:10:59.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd38c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd341035a0 0x7fdd3419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:59.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd38c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd34103f50 0x7fdd3419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:59.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd38c00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 0x7fdd34199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:59.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd38c00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34113590 con 0x7fdd34108680 2024-04-26T20:10:59.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd38c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34113410 con 0x7fdd341035a0 2024-04-26T20:10:59.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd38c00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34113710 con 0x7fdd34103f50 2024-04-26T20:10:59.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd33e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 0x7fdd34199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:59.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd33400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd341035a0 0x7fdd3419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd32a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd34103f50 0x7fdd3419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:59.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd33400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd341035a0 0x7fdd3419f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:59050/0 (socket says 172.21.15.94:59050) 2024-04-26T20:10:59.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd33e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 0x7fdd34199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42800/0 (socket says 172.21.15.94:42800) 2024-04-26T20:10:59.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd32a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd34103f50 0x7fdd3419fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:40578/0 (socket says 172.21.15.94:40578) 2024-04-26T20:10:59.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.946+0000 7fdd33400700 1 -- 172.21.15.94:0/914098704 learned_addr learned my addr 172.21.15.94:0/914098704 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:10:59.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd32a00700 1 -- 172.21.15.94:0/914098704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd341035a0 msgr2=0x7fdd3419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:59.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd32a00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd341035a0 0x7fdd3419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:59.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd32a00700 1 -- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 msgr2=0x7fdd34199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:59.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd32a00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 0x7fdd34199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:59.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd32a00700 1 -- 172.21.15.94:0/914098704 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3419a070 con 0x7fdd34103f50 2024-04-26T20:10:59.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd33e00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 0x7fdd34199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:10:59.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd32a00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd34103f50 0x7fdd3419fc00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000e9c0 tx=0x7fdd2000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:59.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd2be00700 1 -- 172.21.15.94:0/914098704 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2000c9c0 con 0x7fdd34103f50 2024-04-26T20:10:59.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd2be00700 1 -- 172.21.15.94:0/914098704 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd2000cb60 con 0x7fdd34103f50 2024-04-26T20:10:59.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd2be00700 1 -- 172.21.15.94:0/914098704 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20018680 con 0x7fdd34103f50 2024-04-26T20:10:59.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.947+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd3419a360 con 0x7fdd34103f50 2024-04-26T20:10:59.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.948+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd341a6520 con 0x7fdd34103f50 2024-04-26T20:10:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.948+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd34104a80 con 0x7fdd34103f50 2024-04-26T20:10:59.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.951+0000 7fdd2be00700 1 -- 172.21.15.94:0/914098704 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdd20010430 con 0x7fdd34103f50 2024-04-26T20:10:59.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.953+0000 7fdd2be00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd180a3cd0 0x7fdd180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:10:59.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.953+0000 7fdd2be00700 1 -- 172.21.15.94:0/914098704 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fdd200d0bc0 con 0x7fdd34103f50 2024-04-26T20:10:59.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.953+0000 7fdd33400700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd180a3cd0 0x7fdd180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:10:59.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.953+0000 7fdd33400700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd180a3cd0 0x7fdd180a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009d60 tx=0x7fdd24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:10:59.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:57.954+0000 7fdd2be00700 1 -- 172.21.15.94:0/914098704 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdd2009d010 con 0x7fdd34103f50 2024-04-26T20:10:59.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.190+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fdd3402e490 con 0x7fdd34103f50 2024-04-26T20:10:59.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.191+0000 7fdd2be00700 1 -- 172.21.15.94:0/914098704 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdd2009c910 con 0x7fdd34103f50 2024-04-26T20:10:59.803 INFO:teuthology.orchestra.run.smithi094.stdout:141733920855 2024-04-26T20:10:59.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd180a3cd0 msgr2=0x7fdd180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:59.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd180a3cd0 0x7fdd180a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009d60 tx=0x7fdd24009260 comp rx=0 tx=0).stop 2024-04-26T20:10:59.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd34103f50 msgr2=0x7fdd3419fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:10:59.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd34103f50 0x7fdd3419fc00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000e9c0 tx=0x7fdd2000ecd0 comp rx=0 tx=0).stop 2024-04-26T20:10:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 shutdown_connections 2024-04-26T20:10:59.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd34108680 0x7fdd34199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:59.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd180a3cd0 0x7fdd180a6190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:10:59.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd341035a0 0x7fdd3419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:00.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 --2- 172.21.15.94:0/914098704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd34103f50 0x7fdd3419fc00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:00.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 >> 172.21.15.94:0/914098704 conn(0x7fdd340fee70 msgr2=0x7fdd341011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:00.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 shutdown_connections 2024-04-26T20:11:00.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.194+0000 7fdd38c00700 1 -- 172.21.15.94:0/914098704 wait complete. 2024-04-26T20:11:00.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.479+0000 7fd9a5000700 1 -- 172.21.15.94:0/576173806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 msgr2=0x7fd9a0109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:00.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.479+0000 7fd9a5000700 1 --2- 172.21.15.94:0/576173806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 0x7fd9a0109ed0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009990 tx=0x7fd98c009ca0 comp rx=0 tx=0).stop 2024-04-26T20:11:00.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.479+0000 7fd9a5000700 1 -- 172.21.15.94:0/576173806 shutdown_connections 2024-04-26T20:11:00.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.479+0000 7fd9a5000700 1 --2- 172.21.15.94:0/576173806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a010a410 0x7fd9a0116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:00.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.479+0000 7fd9a5000700 1 --2- 172.21.15.94:0/576173806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9a010fa50 0x7fd9a010fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:00.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.479+0000 7fd9a5000700 1 --2- 172.21.15.94:0/576173806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 0x7fd9a0109ed0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:00.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.479+0000 7fd9a5000700 1 -- 172.21.15.94:0/576173806 >> 172.21.15.94:0/576173806 conn(0x7fd9a0076bf0 msgr2=0x7fd9a0108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.479+0000 7fd9a5000700 1 -- 172.21.15.94:0/576173806 shutdown_connections 2024-04-26T20:11:00.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.479+0000 7fd9a5000700 1 -- 172.21.15.94:0/576173806 wait complete. 2024-04-26T20:11:00.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.481+0000 7fd9a5000700 1 Processor -- start 2024-04-26T20:11:00.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.481+0000 7fd9a5000700 1 -- start start 2024-04-26T20:11:00.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.481+0000 7fd9a5000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 0x7fd9a01a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:00.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.481+0000 7fd9a5000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9a010a410 0x7fd9a01a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:00.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.481+0000 7fd9a5000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a010fa50 0x7fd9a01a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:00.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.481+0000 7fd9a5000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0119ce0 con 0x7fd9a0109a50 2024-04-26T20:11:00.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.481+0000 7fd9a5000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0119b60 con 0x7fd9a010fa50 2024-04-26T20:11:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.481+0000 7fd9a5000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0119e60 con 0x7fd9a010a410 2024-04-26T20:11:00.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.481+0000 7fd99de00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 0x7fd9a01a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:00.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99de00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 0x7fd9a01a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42814/0 (socket says 172.21.15.94:42814) 2024-04-26T20:11:00.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99de00700 1 -- 172.21.15.94:0/3746272470 learned_addr learned my addr 172.21.15.94:0/3746272470 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:00.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99e800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a010fa50 0x7fd9a01a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99d400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9a010a410 0x7fd9a01a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:00.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99e800700 1 -- 172.21.15.94:0/3746272470 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9a010a410 msgr2=0x7fd9a01a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99e800700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9a010a410 0x7fd9a01a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:00.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99e800700 1 -- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 msgr2=0x7fd9a01a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:00.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99e800700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 0x7fd9a01a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:00.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99e800700 1 -- 172.21.15.94:0/3746272470 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a01a51c0 con 0x7fd9a010fa50 2024-04-26T20:11:00.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99de00700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 0x7fd9a01a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:00.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99d400700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9a010a410 0x7fd9a01a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:00.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.482+0000 7fd99e800700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a010fa50 0x7fd9a01a4a80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd99800efc0 tx=0x7fd99800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.483+0000 7fd996a00700 1 -- 172.21.15.94:0/3746272470 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9980092e0 con 0x7fd9a010fa50 2024-04-26T20:11:00.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.483+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a01a5450 con 0x7fd9a010fa50 2024-04-26T20:11:00.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.483+0000 7fd996a00700 1 -- 172.21.15.94:0/3746272470 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd998004bb0 con 0x7fd9a010fa50 2024-04-26T20:11:00.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.483+0000 7fd996a00700 1 -- 172.21.15.94:0/3746272470 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd998004020 con 0x7fd9a010fa50 2024-04-26T20:11:00.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.483+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a01acd30 con 0x7fd9a010fa50 2024-04-26T20:11:01.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.484+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a006a110 con 0x7fd9a010fa50 2024-04-26T20:11:01.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.486+0000 7fd996a00700 1 -- 172.21.15.94:0/3746272470 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd99801b030 con 0x7fd9a010fa50 2024-04-26T20:11:01.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.487+0000 7fd996a00700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9840a3cd0 0x7fd9840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.487+0000 7fd996a00700 1 -- 172.21.15.94:0/3746272470 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fd9980d0ea0 con 0x7fd9a010fa50 2024-04-26T20:11:01.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.488+0000 7fd99de00700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9840a3cd0 0x7fd9840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:01.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.488+0000 7fd99de00700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9840a3cd0 0x7fd9840a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd990011550 tx=0x7fd990009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.490+0000 7fd996a00700 1 -- 172.21.15.94:0/3746272470 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd998022020 con 0x7fd9a010fa50 2024-04-26T20:11:01.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.726+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd9a002e490 con 0x7fd9a010fa50 2024-04-26T20:11:01.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.726+0000 7fd996a00700 1 -- 172.21.15.94:0/3746272470 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd998010040 con 0x7fd9a010fa50 2024-04-26T20:11:01.265 INFO:teuthology.orchestra.run.smithi094.stdout:163208757330 2024-04-26T20:11:01.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9840a3cd0 msgr2=0x7fd9840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:01.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9840a3cd0 0x7fd9840a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd990011550 tx=0x7fd990009260 comp rx=0 tx=0).stop 2024-04-26T20:11:01.291 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:58 smithi094 ceph-mon[28050]: pgmap v409: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:01.360 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:58 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1225130471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T20:11:01.370 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:58 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/914098704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T20:11:01.377 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:58 smithi204 ceph-mon[32618]: pgmap v409: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:01.479 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:58 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1225130471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T20:11:01.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:58 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/914098704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T20:11:01.568 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:10:59 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3746272470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T20:11:01.583 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:00 smithi204 ceph-mon[32618]: pgmap v410: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T20:11:01.591 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:58 smithi103 ceph-mon[32635]: pgmap v409: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:01.600 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:58 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1225130471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T20:11:01.612 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:58 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/914098704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T20:11:01.674 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:10:59 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3746272470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T20:11:01.687 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:00 smithi103 ceph-mon[32635]: pgmap v410: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T20:11:01.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a010fa50 msgr2=0x7fd9a01a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:01.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a010fa50 0x7fd9a01a4a80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd99800efc0 tx=0x7fd99800c3a0 comp rx=0 tx=0).stop 2024-04-26T20:11:01.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 shutdown_connections 2024-04-26T20:11:01.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd9840a3cd0 0x7fd9840a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:01.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9a010fa50 0x7fd9a01a4a80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9a0109a50 0x7fd9a01a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:01.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 --2- 172.21.15.94:0/3746272470 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9a010a410 0x7fd9a01a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:01.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.729+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 >> 172.21.15.94:0/3746272470 conn(0x7fd9a0076bf0 msgr2=0x7fd9a00772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:01.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.730+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 shutdown_connections 2024-04-26T20:11:02.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:10:58.730+0000 7fd9a5000700 1 -- 172.21.15.94:0/3746272470 wait complete. 2024-04-26T20:11:02.071 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920849 got 141733920855 for osd.4 2024-04-26T20:11:02.079 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:10:59 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3746272470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T20:11:02.087 DEBUG:teuthology.parallel:result is None 2024-04-26T20:11:02.102 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:01 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/659982964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T20:11:02.131 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645357 got 309237645364 for osd.10 2024-04-26T20:11:02.157 DEBUG:teuthology.parallel:result is None 2024-04-26T20:11:02.168 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:01 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/659982964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T20:11:02.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.775+0000 7f2513200700 1 -- 172.21.15.94:0/2641454380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f250c105640 msgr2=0x7f250c105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:02.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.775+0000 7f2513200700 1 --2- 172.21.15.94:0/2641454380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f250c105640 0x7f250c105ac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f24f80098f0 tx=0x7f24f8009c00 comp rx=0 tx=0).stop 2024-04-26T20:11:02.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.775+0000 7f2513200700 1 -- 172.21.15.94:0/2641454380 shutdown_connections 2024-04-26T20:11:02.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.775+0000 7f2513200700 1 --2- 172.21.15.94:0/2641454380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c106000 0x7f250c112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.775+0000 7f2513200700 1 --2- 172.21.15.94:0/2641454380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c10b640 0x7f250c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.775+0000 7f2513200700 1 --2- 172.21.15.94:0/2641454380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f250c105640 0x7f250c105ac0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:02.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.775+0000 7f2513200700 1 -- 172.21.15.94:0/2641454380 >> 172.21.15.94:0/2641454380 conn(0x7f250c101130 msgr2=0x7f250c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:02.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.775+0000 7f2513200700 1 -- 172.21.15.94:0/2641454380 shutdown_connections 2024-04-26T20:11:02.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.775+0000 7f2513200700 1 -- 172.21.15.94:0/2641454380 wait complete. 2024-04-26T20:11:02.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.777+0000 7f2513200700 1 Processor -- start 2024-04-26T20:11:02.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.777+0000 7f2513200700 1 -- start start 2024-04-26T20:11:02.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.777+0000 7f2513200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c105640 0x7f250c19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:02.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.777+0000 7f2513200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c106000 0x7f250c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:02.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.777+0000 7f2513200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f250c10b640 0x7f250c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:02.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.777+0000 7f2513200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f250c1158d0 con 0x7f250c10b640 2024-04-26T20:11:02.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.777+0000 7f2513200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f250c115750 con 0x7f250c106000 2024-04-26T20:11:02.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.777+0000 7f2513200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f250c115a50 con 0x7f250c105640 2024-04-26T20:11:02.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f250be00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c105640 0x7f250c19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:02.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f250b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c106000 0x7f250c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:02.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f250be00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c105640 0x7f250c19bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:40628/0 (socket says 172.21.15.94:40628) 2024-04-26T20:11:02.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f250be00700 1 -- 172.21.15.94:0/659982964 learned_addr learned my addr 172.21.15.94:0/659982964 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:02.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f2511200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f250c10b640 0x7f250c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:02.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f2511200700 1 -- 172.21.15.94:0/659982964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c105640 msgr2=0x7f250c19bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:02.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f2511200700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c105640 0x7f250c19bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:02.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f2511200700 1 -- 172.21.15.94:0/659982964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c106000 msgr2=0x7f250c19c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:02.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f2511200700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c106000 0x7f250c19c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f2511200700 1 -- 172.21.15.94:0/659982964 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f250c1a0d90 con 0x7f250c10b640 2024-04-26T20:11:02.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.778+0000 7f250b400700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c106000 0x7f250c19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:02.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.779+0000 7f2511200700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f250c10b640 0x7f250c1a0650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2504009cd0 tx=0x7f250400c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:02.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.779+0000 7f2508c00700 1 -- 172.21.15.94:0/659982964 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250400cd10 con 0x7f250c10b640 2024-04-26T20:11:02.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.779+0000 7f2508c00700 1 -- 172.21.15.94:0/659982964 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2504003a00 con 0x7f250c10b640 2024-04-26T20:11:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.779+0000 7f2508c00700 1 -- 172.21.15.94:0/659982964 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2504004520 con 0x7f250c10b640 2024-04-26T20:11:02.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.779+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f250c1a1020 con 0x7f250c10b640 2024-04-26T20:11:02.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.779+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f250c1a8900 con 0x7f250c10b640 2024-04-26T20:11:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.780+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f250c06a110 con 0x7f250c10b640 2024-04-26T20:11:02.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.782+0000 7f2508c00700 1 -- 172.21.15.94:0/659982964 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f250401a020 con 0x7f250c10b640 2024-04-26T20:11:03.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.784+0000 7f2508c00700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f24f00a3d50 0x7f24f00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.784+0000 7f2508c00700 1 -- 172.21.15.94:0/659982964 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f25040d0220 con 0x7f250c10b640 2024-04-26T20:11:03.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.784+0000 7f250be00700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f24f00a3d50 0x7f24f00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:03.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.785+0000 7f250be00700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f24f00a3d50 0x7f24f00a6210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009680 tx=0x7f24fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:03.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:00.789+0000 7f2508c00700 1 -- 172.21.15.94:0/659982964 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2504010040 con 0x7f250c10b640 2024-04-26T20:11:03.065 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:00 smithi094 ceph-mon[28050]: pgmap v410: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T20:11:03.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.032+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f250c02e490 con 0x7f250c10b640 2024-04-26T20:11:03.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.033+0000 7f2508c00700 1 -- 172.21.15.94:0/659982964 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f250409c670 con 0x7f250c10b640 2024-04-26T20:11:03.229 INFO:teuthology.orchestra.run.smithi094.stdout:197568495692 2024-04-26T20:11:03.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f24f00a3d50 msgr2=0x7f24f00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:03.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f24f00a3d50 0x7f24f00a6210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009680 tx=0x7f24fc009260 comp rx=0 tx=0).stop 2024-04-26T20:11:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f250c10b640 msgr2=0x7f250c1a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:03.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f250c10b640 0x7f250c1a0650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2504009cd0 tx=0x7f250400c620 comp rx=0 tx=0).stop 2024-04-26T20:11:03.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 shutdown_connections 2024-04-26T20:11:03.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f250c10b640 0x7f250c1a0650 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c105640 0x7f250c19bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:03.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f24f00a3d50 0x7f24f00a6210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:03.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 --2- 172.21.15.94:0/659982964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c106000 0x7f250c19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:03.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 >> 172.21.15.94:0/659982964 conn(0x7f250c101130 msgr2=0x7f250c1019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:03.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 shutdown_connections 2024-04-26T20:11:03.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.036+0000 7f2513200700 1 -- 172.21.15.94:0/659982964 wait complete. 2024-04-26T20:11:03.452 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:02 smithi103 ceph-mon[32635]: pgmap v411: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:03.459 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:02 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3424746136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T20:11:03.512 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:02 smithi204 ceph-mon[32618]: pgmap v411: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:03.556 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:02 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3424746136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T20:11:03.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.863+0000 7f66abe00700 1 -- 172.21.15.94:0/3417918710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a4109490 msgr2=0x7f66a4109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:03.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.863+0000 7f66abe00700 1 --2- 172.21.15.94:0/3417918710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a4109490 0x7f66a4109870 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6698009990 tx=0x7f6698009ca0 comp rx=0 tx=0).stop 2024-04-26T20:11:03.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.863+0000 7f66abe00700 1 -- 172.21.15.94:0/3417918710 shutdown_connections 2024-04-26T20:11:03.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.863+0000 7f66abe00700 1 --2- 172.21.15.94:0/3417918710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a406c360 0x7f66a41102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:03.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.863+0000 7f66abe00700 1 --2- 172.21.15.94:0/3417918710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a4109490 0x7f66a4109870 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:03.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.863+0000 7f66abe00700 1 --2- 172.21.15.94:0/3417918710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a406b9c0 0x7f66a406be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.863+0000 7f66abe00700 1 -- 172.21.15.94:0/3417918710 >> 172.21.15.94:0/3417918710 conn(0x7f66a4079290 msgr2=0x7f66a40796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:03.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.863+0000 7f66abe00700 1 -- 172.21.15.94:0/3417918710 shutdown_connections 2024-04-26T20:11:03.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.863+0000 7f66abe00700 1 -- 172.21.15.94:0/3417918710 wait complete. 2024-04-26T20:11:03.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66abe00700 1 Processor -- start 2024-04-26T20:11:03.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66abe00700 1 -- start start 2024-04-26T20:11:03.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66abe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a406b9c0 0x7f66a41023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:03.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66abe00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a406c360 0x7f66a4102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:03.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66abe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a4109490 0x7f66a4102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:03.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66abe00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f66a4113580 con 0x7f66a406b9c0 2024-04-26T20:11:03.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66abe00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f66a4113400 con 0x7f66a4109490 2024-04-26T20:11:03.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66abe00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f66a4113700 con 0x7f66a406c360 2024-04-26T20:11:03.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66a9400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a406b9c0 0x7f66a41023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:03.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66a9400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a406b9c0 0x7f66a41023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47676/0 (socket says 172.21.15.94:47676) 2024-04-26T20:11:03.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66a9400700 1 -- 172.21.15.94:0/3424746136 learned_addr learned my addr 172.21.15.94:0/3424746136 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:04.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66a8a00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a406c360 0x7f66a4102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:04.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.865+0000 7f66a9e00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a4109490 0x7f66a4102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:04.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.866+0000 7f66a9400700 1 -- 172.21.15.94:0/3424746136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a406c360 msgr2=0x7f66a4102920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:04.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.866+0000 7f66a9400700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a406c360 0x7f66a4102920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:04.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.866+0000 7f66a9400700 1 -- 172.21.15.94:0/3424746136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a4109490 msgr2=0x7f66a4102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:04.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.866+0000 7f66a9400700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a4109490 0x7f66a4102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:04.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.866+0000 7f66a9400700 1 -- 172.21.15.94:0/3424746136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a41a8020 con 0x7f66a406b9c0 2024-04-26T20:11:04.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.866+0000 7f66a9400700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a406b9c0 0x7f66a41023e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f66980057c0 tx=0x7f6698009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:04.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.867+0000 7f66a2000700 1 -- 172.21.15.94:0/3424746136 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698026070 con 0x7f66a406b9c0 2024-04-26T20:11:04.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.867+0000 7f66a2000700 1 -- 172.21.15.94:0/3424746136 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6698021410 con 0x7f66a406b9c0 2024-04-26T20:11:04.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.867+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a41a82b0 con 0x7f66a406b9c0 2024-04-26T20:11:04.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.867+0000 7f66a2000700 1 -- 172.21.15.94:0/3424746136 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698017650 con 0x7f66a406b9c0 2024-04-26T20:11:04.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.867+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a41a8790 con 0x7f66a406b9c0 2024-04-26T20:11:04.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.868+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66a406a110 con 0x7f66a406b9c0 2024-04-26T20:11:04.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.870+0000 7f66a2000700 1 -- 172.21.15.94:0/3424746136 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6698029070 con 0x7f66a406b9c0 2024-04-26T20:11:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.871+0000 7f66a2000700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f668c0a3cd0 0x7f668c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:04.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.871+0000 7f66a2000700 1 -- 172.21.15.94:0/3424746136 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f66980d1660 con 0x7f66a406b9c0 2024-04-26T20:11:04.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.872+0000 7f66a8a00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f668c0a3cd0 0x7f668c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:04.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.872+0000 7f66a8a00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f668c0a3cd0 0x7f668c0a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f66a40799e0 tx=0x7f6690009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:04.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:01.876+0000 7f66a2000700 1 -- 172.21.15.94:0/3424746136 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f669805e060 con 0x7f66a406b9c0 2024-04-26T20:11:04.543 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:01 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/659982964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T20:11:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.123+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f66a402e490 con 0x7f66a406b9c0 2024-04-26T20:11:04.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.124+0000 7f66a2000700 1 -- 172.21.15.94:0/3424746136 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6698013070 con 0x7f66a406b9c0 2024-04-26T20:11:04.568 INFO:teuthology.orchestra.run.smithi094.stdout:253403070529 2024-04-26T20:11:04.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f668c0a3cd0 msgr2=0x7f668c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:04.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f668c0a3cd0 0x7f668c0a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f66a40799e0 tx=0x7f6690009260 comp rx=0 tx=0).stop 2024-04-26T20:11:04.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a406b9c0 msgr2=0x7f66a41023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:04.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a406b9c0 0x7f66a41023e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f66980057c0 tx=0x7f6698009580 comp rx=0 tx=0).stop 2024-04-26T20:11:04.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 shutdown_connections 2024-04-26T20:11:04.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f668c0a3cd0 0x7f668c0a6190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:04.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66a4109490 0x7f66a4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:04.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66a406b9c0 0x7f66a41023e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:04.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 --2- 172.21.15.94:0/3424746136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a406c360 0x7f66a4102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:04.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 >> 172.21.15.94:0/3424746136 conn(0x7f66a4079290 msgr2=0x7f66a4100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:04.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 shutdown_connections 2024-04-26T20:11:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:02.127+0000 7f66abe00700 1 -- 172.21.15.94:0/3424746136 wait complete. 2024-04-26T20:11:04.953 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495685 got 197568495692 for osd.6 2024-04-26T20:11:04.965 DEBUG:teuthology.parallel:result is None 2024-04-26T20:11:04.973 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757325 got 163208757330 for osd.5 2024-04-26T20:11:04.981 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:02 smithi094 ceph-mon[28050]: pgmap v411: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:04.993 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:02 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3424746136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T20:11:05.003 DEBUG:teuthology.parallel:result is None 2024-04-26T20:11:05.014 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070523 got 253403070529 for osd.8 2024-04-26T20:11:05.028 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:04 smithi094 ceph-mon[28050]: pgmap v412: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:05.041 DEBUG:teuthology.parallel:result is None 2024-04-26T20:11:05.053 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T20:11:05.062 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T20:11:05.125 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:04 smithi103 ceph-mon[32635]: pgmap v412: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:05.201 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:04 smithi204 ceph-mon[32618]: pgmap v412: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:05.574 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:11:06.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:06 smithi094 ceph-mon[28050]: pgmap v413: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:07.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:06 smithi103 ceph-mon[32635]: pgmap v413: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:07.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:06 smithi204 ceph-mon[32618]: pgmap v413: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:08.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.006+0000 7f02cf600700 1 -- 172.21.15.94:0/745846815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02c8103470 msgr2=0x7f02c81038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:08.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.006+0000 7f02cf600700 1 --2- 172.21.15.94:0/745846815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02c8103470 0x7f02c81038f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f02b00098f0 tx=0x7f02b0009c00 comp rx=0 tx=0).stop 2024-04-26T20:11:08.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.006+0000 7f02cf600700 1 -- 172.21.15.94:0/745846815 shutdown_connections 2024-04-26T20:11:08.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.006+0000 7f02cf600700 1 --2- 172.21.15.94:0/745846815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02c8103e30 0x7f02c8110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:08.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.006+0000 7f02cf600700 1 --2- 172.21.15.94:0/745846815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c81094c0 0x7f02c81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:08.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.006+0000 7f02cf600700 1 --2- 172.21.15.94:0/745846815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02c8103470 0x7f02c81038f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:08.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.006+0000 7f02cf600700 1 -- 172.21.15.94:0/745846815 >> 172.21.15.94:0/745846815 conn(0x7f02c807a580 msgr2=0x7f02c807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:08.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.006+0000 7f02cf600700 1 -- 172.21.15.94:0/745846815 shutdown_connections 2024-04-26T20:11:08.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.006+0000 7f02cf600700 1 -- 172.21.15.94:0/745846815 wait complete. 2024-04-26T20:11:08.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.007+0000 7f02cf600700 1 Processor -- start 2024-04-26T20:11:08.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02cf600700 1 -- start start 2024-04-26T20:11:08.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02cf600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02c8103470 0x7f02c819be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02cf600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02c8103e30 0x7f02c819c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:08.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02cf600700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c81094c0 0x7f02c81a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:08.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02cf600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f02c8113630 con 0x7f02c8103470 2024-04-26T20:11:08.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02cf600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81134b0 con 0x7f02c8103e30 2024-04-26T20:11:08.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02cf600700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81137b0 con 0x7f02c81094c0 2024-04-26T20:11:08.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02ccc00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02c8103470 0x7f02c819be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:08.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02ccc00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02c8103470 0x7f02c819be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47692/0 (socket says 172.21.15.94:47692) 2024-04-26T20:11:08.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02bfe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02c8103e30 0x7f02c819c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:08.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02ccc00700 1 -- 172.21.15.94:0/382655240 learned_addr learned my addr 172.21.15.94:0/382655240 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:08.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.008+0000 7f02cd600700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c81094c0 0x7f02c81a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:08.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.009+0000 7f02ccc00700 1 -- 172.21.15.94:0/382655240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c81094c0 msgr2=0x7f02c81a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:08.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.009+0000 7f02ccc00700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c81094c0 0x7f02c81a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:08.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.009+0000 7f02ccc00700 1 -- 172.21.15.94:0/382655240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02c8103e30 msgr2=0x7f02c819c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:08.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.009+0000 7f02ccc00700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02c8103e30 0x7f02c819c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:08.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.009+0000 7f02ccc00700 1 -- 172.21.15.94:0/382655240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c81a0e30 con 0x7f02c8103470 2024-04-26T20:11:08.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.009+0000 7f02bfe00700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02c8103e30 0x7f02c819c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:08.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.009+0000 7f02ccc00700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02c8103470 0x7f02c819be20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f02b800e9c0 tx=0x7f02b800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:08.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.009+0000 7f02c6000700 1 -- 172.21.15.94:0/382655240 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b800cbd0 con 0x7f02c8103470 2024-04-26T20:11:08.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.010+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c81a1120 con 0x7f02c8103470 2024-04-26T20:11:08.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.010+0000 7f02c6000700 1 -- 172.21.15.94:0/382655240 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02b800ee60 con 0x7f02c8103470 2024-04-26T20:11:08.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.010+0000 7f02c6000700 1 -- 172.21.15.94:0/382655240 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b80055c0 con 0x7f02c8103470 2024-04-26T20:11:08.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.010+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c81a89a0 con 0x7f02c8103470 2024-04-26T20:11:08.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.011+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c806a110 con 0x7f02c8103470 2024-04-26T20:11:08.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.013+0000 7f02c6000700 1 -- 172.21.15.94:0/382655240 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f02b8004750 con 0x7f02c8103470 2024-04-26T20:11:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.014+0000 7f02c6000700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02ac0a3cd0 0x7f02ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:08.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.015+0000 7f02c6000700 1 -- 172.21.15.94:0/382655240 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f02b80d0de0 con 0x7f02c8103470 2024-04-26T20:11:08.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.015+0000 7f02bfe00700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02ac0a3cd0 0x7f02ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:08.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.015+0000 7f02bfe00700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02ac0a3cd0 0x7f02ac0a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f02b00098c0 tx=0x7f02b0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:08.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.017+0000 7f02c6000700 1 -- 172.21.15.94:0/382655240 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f02b8054060 con 0x7f02c8103470 2024-04-26T20:11:08.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.240+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f02c819d180 con 0x7f02ac0a3cd0 2024-04-26T20:11:08.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.256+0000 7f02c6000700 1 -- 172.21.15.94:0/382655240 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340381 (secure 0 0 0) 0x7f02c819d180 con 0x7f02ac0a3cd0 2024-04-26T20:11:08.858 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:11:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.263+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02ac0a3cd0 msgr2=0x7f02ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.263+0000 7f02cf600700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02ac0a3cd0 0x7f02ac0a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f02b00098c0 tx=0x7f02b0019040 comp rx=0 tx=0).stop 2024-04-26T20:11:08.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02c8103470 msgr2=0x7f02c819be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:09.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02c8103470 0x7f02c819be20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f02b800e9c0 tx=0x7f02b800ecd0 comp rx=0 tx=0).stop 2024-04-26T20:11:09.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 shutdown_connections 2024-04-26T20:11:09.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c81094c0 0x7f02c81a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:09.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02c8103470 0x7f02c819be20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:09.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02ac0a3cd0 0x7f02ac0a6190 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 --2- 172.21.15.94:0/382655240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02c8103e30 0x7f02c819c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 >> 172.21.15.94:0/382655240 conn(0x7f02c807a580 msgr2=0x7f02c810ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:09.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 shutdown_connections 2024-04-26T20:11:09.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:08.264+0000 7f02cf600700 1 -- 172.21.15.94:0/382655240 wait complete. 2024-04-26T20:11:09.157 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-26T20:11:09.163 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:08 smithi094 ceph-mon[28050]: pgmap v414: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:09.173 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:08 smithi204 ceph-mon[32618]: pgmap v414: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:09.179 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:08 smithi103 ceph-mon[32635]: pgmap v414: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:09.205 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-04-26T20:11:07.629749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651714,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":747,"num_read_kb":489,"num_write":352,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1182331,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":397448,"kb_used_data":14936,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124544376,"statfs":{"total":1151940427776,"available":1151533441024,"internally_reserved":0,"allocated":15294464,"data_stored":8469368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":391492924},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004687"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766125+0000","last_change":"2024-04-26T20:08:17.653922+0000","last_active":"2024-04-26T20:10:13.766125+0000","last_peered":"2024-04-26T20:10:13.766125+0000","last_clean":"2024-04-26T20:10:13.766125+0000","last_became_active":"2024-04-26T20:08:17.653486+0000","last_became_peered":"2024-04-26T20:08:17.653486+0000","last_unstale":"2024-04-26T20:10:13.766125+0000","last_undegraded":"2024-04-26T20:10:13.766125+0000","last_fullsized":"2024-04-26T20:10:13.766125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:05.034829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350663+0000","last_change":"2024-04-26T20:08:17.650126+0000","last_active":"2024-04-26T20:10:14.350663+0000","last_peered":"2024-04-26T20:10:14.350663+0000","last_clean":"2024-04-26T20:10:14.350663+0000","last_became_active":"2024-04-26T20:08:17.649727+0000","last_became_peered":"2024-04-26T20:08:17.649727+0000","last_unstale":"2024-04-26T20:10:14.350663+0000","last_undegraded":"2024-04-26T20:10:14.350663+0000","last_fullsized":"2024-04-26T20:10:14.350663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:11:28.255286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350429+0000","last_change":"2024-04-26T20:08:17.646243+0000","last_active":"2024-04-26T20:10:14.350429+0000","last_peered":"2024-04-26T20:10:14.350429+0000","last_clean":"2024-04-26T20:10:14.350429+0000","last_became_active":"2024-04-26T20:08:17.645677+0000","last_became_peered":"2024-04-26T20:08:17.645677+0000","last_unstale":"2024-04-26T20:10:14.350429+0000","last_undegraded":"2024-04-26T20:10:14.350429+0000","last_fullsized":"2024-04-26T20:10:14.350429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:31.698329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347798+0000","last_change":"2024-04-26T20:08:17.639845+0000","last_active":"2024-04-26T20:10:14.347798+0000","last_peered":"2024-04-26T20:10:14.347798+0000","last_clean":"2024-04-26T20:10:14.347798+0000","last_became_active":"2024-04-26T20:08:17.639465+0000","last_became_peered":"2024-04-26T20:08:17.639465+0000","last_unstale":"2024-04-26T20:10:14.347798+0000","last_undegraded":"2024-04-26T20:10:14.347798+0000","last_fullsized":"2024-04-26T20:10:14.347798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:05.717196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768688+0000","last_change":"2024-04-26T20:08:17.660459+0000","last_active":"2024-04-26T20:10:13.768688+0000","last_peered":"2024-04-26T20:10:13.768688+0000","last_clean":"2024-04-26T20:10:13.768688+0000","last_became_active":"2024-04-26T20:08:17.659927+0000","last_became_peered":"2024-04-26T20:08:17.659927+0000","last_unstale":"2024-04-26T20:10:13.768688+0000","last_undegraded":"2024-04-26T20:10:13.768688+0000","last_fullsized":"2024-04-26T20:10:13.768688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:10.062073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763490+0000","last_change":"2024-04-26T20:09:13.279825+0000","last_active":"2024-04-26T20:10:13.763490+0000","last_peered":"2024-04-26T20:10:13.763490+0000","last_clean":"2024-04-26T20:10:13.763490+0000","last_became_active":"2024-04-26T20:09:13.279464+0000","last_became_peered":"2024-04-26T20:09:13.279464+0000","last_unstale":"2024-04-26T20:10:13.763490+0000","last_undegraded":"2024-04-26T20:10:13.763490+0000","last_fullsized":"2024-04-26T20:10:13.763490+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:41.687987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768026+0000","last_change":"2024-04-26T20:08:17.648806+0000","last_active":"2024-04-26T20:10:13.768026+0000","last_peered":"2024-04-26T20:10:13.768026+0000","last_clean":"2024-04-26T20:10:13.768026+0000","last_became_active":"2024-04-26T20:08:17.648552+0000","last_became_peered":"2024-04-26T20:08:17.648552+0000","last_unstale":"2024-04-26T20:10:13.768026+0000","last_undegraded":"2024-04-26T20:10:13.768026+0000","last_fullsized":"2024-04-26T20:10:13.768026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:43.282610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763953+0000","last_change":"2024-04-26T20:08:17.663479+0000","last_active":"2024-04-26T20:10:13.763953+0000","last_peered":"2024-04-26T20:10:13.763953+0000","last_clean":"2024-04-26T20:10:13.763953+0000","last_became_active":"2024-04-26T20:08:17.663043+0000","last_became_peered":"2024-04-26T20:08:17.663043+0000","last_unstale":"2024-04-26T20:10:13.763953+0000","last_undegraded":"2024-04-26T20:10:13.763953+0000","last_fullsized":"2024-04-26T20:10:13.763953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:21:36.518513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930272+0000","last_change":"2024-04-26T20:08:17.637978+0000","last_active":"2024-04-26T20:10:13.930272+0000","last_peered":"2024-04-26T20:10:13.930272+0000","last_clean":"2024-04-26T20:10:13.930272+0000","last_became_active":"2024-04-26T20:08:17.637458+0000","last_became_peered":"2024-04-26T20:08:17.637458+0000","last_unstale":"2024-04-26T20:10:13.930272+0000","last_undegraded":"2024-04-26T20:10:13.930272+0000","last_fullsized":"2024-04-26T20:10:13.930272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:20:01.232741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763731+0000","last_change":"2024-04-26T20:08:17.639819+0000","last_active":"2024-04-26T20:10:13.763731+0000","last_peered":"2024-04-26T20:10:13.763731+0000","last_clean":"2024-04-26T20:10:13.763731+0000","last_became_active":"2024-04-26T20:08:17.639479+0000","last_became_peered":"2024-04-26T20:08:17.639479+0000","last_unstale":"2024-04-26T20:10:13.763731+0000","last_undegraded":"2024-04-26T20:10:13.763731+0000","last_fullsized":"2024-04-26T20:10:13.763731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:44.536585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767910+0000","last_change":"2024-04-26T20:08:17.642533+0000","last_active":"2024-04-26T20:10:13.767910+0000","last_peered":"2024-04-26T20:10:13.767910+0000","last_clean":"2024-04-26T20:10:13.767910+0000","last_became_active":"2024-04-26T20:08:17.642198+0000","last_became_peered":"2024-04-26T20:08:17.642198+0000","last_unstale":"2024-04-26T20:10:13.767910+0000","last_undegraded":"2024-04-26T20:10:13.767910+0000","last_fullsized":"2024-04-26T20:10:13.767910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:16:21.200183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347765+0000","last_change":"2024-04-26T20:08:17.644646+0000","last_active":"2024-04-26T20:10:14.347765+0000","last_peered":"2024-04-26T20:10:14.347765+0000","last_clean":"2024-04-26T20:10:14.347765+0000","last_became_active":"2024-04-26T20:08:17.644314+0000","last_became_peered":"2024-04-26T20:08:17.644314+0000","last_unstale":"2024-04-26T20:10:14.347765+0000","last_undegraded":"2024-04-26T20:10:14.347765+0000","last_fullsized":"2024-04-26T20:10:14.347765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:55:00.003120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767886+0000","last_change":"2024-04-26T20:08:17.657118+0000","last_active":"2024-04-26T20:10:13.767886+0000","last_peered":"2024-04-26T20:10:13.767886+0000","last_clean":"2024-04-26T20:10:13.767886+0000","last_became_active":"2024-04-26T20:08:17.656256+0000","last_became_peered":"2024-04-26T20:08:17.656256+0000","last_unstale":"2024-04-26T20:10:13.767886+0000","last_undegraded":"2024-04-26T20:10:13.767886+0000","last_fullsized":"2024-04-26T20:10:13.767886+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:09.133475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.348535+0000","last_change":"2024-04-26T20:08:17.658234+0000","last_active":"2024-04-26T20:10:14.348535+0000","last_peered":"2024-04-26T20:10:14.348535+0000","last_clean":"2024-04-26T20:10:14.348535+0000","last_became_active":"2024-04-26T20:08:17.651714+0000","last_became_peered":"2024-04-26T20:08:17.651714+0000","last_unstale":"2024-04-26T20:10:14.348535+0000","last_undegraded":"2024-04-26T20:10:14.348535+0000","last_fullsized":"2024-04-26T20:10:14.348535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:30.630320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.348612+0000","last_change":"2024-04-26T20:08:17.647898+0000","last_active":"2024-04-26T20:10:14.348612+0000","last_peered":"2024-04-26T20:10:14.348612+0000","last_clean":"2024-04-26T20:10:14.348612+0000","last_became_active":"2024-04-26T20:08:17.647463+0000","last_became_peered":"2024-04-26T20:08:17.647463+0000","last_unstale":"2024-04-26T20:10:14.348612+0000","last_undegraded":"2024-04-26T20:10:14.348612+0000","last_fullsized":"2024-04-26T20:10:14.348612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:24.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.929767+0000","last_change":"2024-04-26T20:08:17.639518+0000","last_active":"2024-04-26T20:10:13.929767+0000","last_peered":"2024-04-26T20:10:13.929767+0000","last_clean":"2024-04-26T20:10:13.929767+0000","last_became_active":"2024-04-26T20:08:17.639183+0000","last_became_peered":"2024-04-26T20:08:17.639183+0000","last_unstale":"2024-04-26T20:10:13.929767+0000","last_undegraded":"2024-04-26T20:10:13.929767+0000","last_fullsized":"2024-04-26T20:10:13.929767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:41:12.893323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765922+0000","last_change":"2024-04-26T20:08:17.653956+0000","last_active":"2024-04-26T20:10:13.765922+0000","last_peered":"2024-04-26T20:10:13.765922+0000","last_clean":"2024-04-26T20:10:13.765922+0000","last_became_active":"2024-04-26T20:08:17.653576+0000","last_became_peered":"2024-04-26T20:08:17.653576+0000","last_unstale":"2024-04-26T20:10:13.765922+0000","last_undegraded":"2024-04-26T20:10:13.765922+0000","last_fullsized":"2024-04-26T20:10:13.765922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:35:09.538194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770895+0000","last_change":"2024-04-26T20:08:17.649500+0000","last_active":"2024-04-26T20:10:13.770895+0000","last_peered":"2024-04-26T20:10:13.770895+0000","last_clean":"2024-04-26T20:10:13.770895+0000","last_became_active":"2024-04-26T20:08:17.649062+0000","last_became_peered":"2024-04-26T20:08:17.649062+0000","last_unstale":"2024-04-26T20:10:13.770895+0000","last_undegraded":"2024-04-26T20:10:13.770895+0000","last_fullsized":"2024-04-26T20:10:13.770895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:24.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764212+0000","last_change":"2024-04-26T20:08:17.666440+0000","last_active":"2024-04-26T20:10:13.764212+0000","last_peered":"2024-04-26T20:10:13.764212+0000","last_clean":"2024-04-26T20:10:13.764212+0000","last_became_active":"2024-04-26T20:08:17.666024+0000","last_became_peered":"2024-04-26T20:08:17.666024+0000","last_unstale":"2024-04-26T20:10:13.764212+0000","last_undegraded":"2024-04-26T20:10:13.764212+0000","last_fullsized":"2024-04-26T20:10:13.764212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:55.356424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763679+0000","last_change":"2024-04-26T20:08:25.700728+0000","last_active":"2024-04-26T20:10:13.763679+0000","last_peered":"2024-04-26T20:10:13.763679+0000","last_clean":"2024-04-26T20:10:13.763679+0000","last_became_active":"2024-04-26T20:08:25.700339+0000","last_became_peered":"2024-04-26T20:08:25.700339+0000","last_unstale":"2024-04-26T20:10:13.763679+0000","last_undegraded":"2024-04-26T20:10:13.763679+0000","last_fullsized":"2024-04-26T20:10:13.763679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:53.581083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'9","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.778543+0000","last_change":"2024-04-26T20:10:13.778038+0000","last_active":"2024-04-26T20:10:13.778543+0000","last_peered":"2024-04-26T20:10:13.778543+0000","last_clean":"2024-04-26T20:10:13.778543+0000","last_became_active":"2024-04-26T20:10:13.772007+0000","last_became_peered":"2024-04-26T20:10:13.772007+0000","last_unstale":"2024-04-26T20:10:13.778543+0000","last_undegraded":"2024-04-26T20:10:13.778543+0000","last_fullsized":"2024-04-26T20:10:13.778543+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:05:26.873150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764986+0000","last_change":"2024-04-26T20:08:17.657637+0000","last_active":"2024-04-26T20:10:13.764986+0000","last_peered":"2024-04-26T20:10:13.764986+0000","last_clean":"2024-04-26T20:10:13.764986+0000","last_became_active":"2024-04-26T20:08:17.657290+0000","last_became_peered":"2024-04-26T20:08:17.657290+0000","last_unstale":"2024-04-26T20:10:13.764986+0000","last_undegraded":"2024-04-26T20:10:13.764986+0000","last_fullsized":"2024-04-26T20:10:13.764986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:25.270989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.346942+0000","last_change":"2024-04-26T20:09:13.318084+0000","last_active":"2024-04-26T20:10:14.346942+0000","last_peered":"2024-04-26T20:10:14.346942+0000","last_clean":"2024-04-26T20:10:14.346942+0000","last_became_active":"2024-04-26T20:09:13.317689+0000","last_became_peered":"2024-04-26T20:09:13.317689+0000","last_unstale":"2024-04-26T20:10:14.346942+0000","last_undegraded":"2024-04-26T20:10:14.346942+0000","last_fullsized":"2024-04-26T20:10:14.346942+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:48.182186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"94'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765720+0000","last_change":"2024-04-26T20:08:16.128717+0000","last_active":"2024-04-26T20:10:13.765720+0000","last_peered":"2024-04-26T20:10:13.765720+0000","last_clean":"2024-04-26T20:10:13.765720+0000","last_became_active":"2024-04-26T20:08:16.127738+0000","last_became_peered":"2024-04-26T20:08:16.127738+0000","last_unstale":"2024-04-26T20:10:13.765720+0000","last_undegraded":"2024-04-26T20:10:13.765720+0000","last_fullsized":"2024-04-26T20:10:13.765720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:01:26.343914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350943+0000","last_change":"2024-04-26T20:08:17.659613+0000","last_active":"2024-04-26T20:10:14.350943+0000","last_peered":"2024-04-26T20:10:14.350943+0000","last_clean":"2024-04-26T20:10:14.350943+0000","last_became_active":"2024-04-26T20:08:17.658750+0000","last_became_peered":"2024-04-26T20:08:17.658750+0000","last_unstale":"2024-04-26T20:10:14.350943+0000","last_undegraded":"2024-04-26T20:10:14.350943+0000","last_fullsized":"2024-04-26T20:10:14.350943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:59:08.724656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347077+0000","last_change":"2024-04-26T20:08:25.709115+0000","last_active":"2024-04-26T20:10:14.347077+0000","last_peered":"2024-04-26T20:10:14.347077+0000","last_clean":"2024-04-26T20:10:14.347077+0000","last_became_active":"2024-04-26T20:08:25.708552+0000","last_became_peered":"2024-04-26T20:08:25.708552+0000","last_unstale":"2024-04-26T20:10:14.347077+0000","last_undegraded":"2024-04-26T20:10:14.347077+0000","last_fullsized":"2024-04-26T20:10:14.347077+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:20.735727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"94'4","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768027+0000","last_change":"2024-04-26T20:08:15.624441+0000","last_active":"2024-04-26T20:10:13.768027+0000","last_peered":"2024-04-26T20:10:13.768027+0000","last_clean":"2024-04-26T20:10:13.768027+0000","last_became_active":"2024-04-26T20:08:15.623999+0000","last_became_peered":"2024-04-26T20:08:15.623999+0000","last_unstale":"2024-04-26T20:10:13.768027+0000","last_undegraded":"2024-04-26T20:10:13.768027+0000","last_fullsized":"2024-04-26T20:10:13.768027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:34.276360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930010+0000","last_change":"2024-04-26T20:08:17.656088+0000","last_active":"2024-04-26T20:10:13.930010+0000","last_peered":"2024-04-26T20:10:13.930010+0000","last_clean":"2024-04-26T20:10:13.930010+0000","last_became_active":"2024-04-26T20:08:17.655736+0000","last_became_peered":"2024-04-26T20:08:17.655736+0000","last_unstale":"2024-04-26T20:10:13.930010+0000","last_undegraded":"2024-04-26T20:10:13.930010+0000","last_fullsized":"2024-04-26T20:10:13.930010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:29.187893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.346963+0000","last_change":"2024-04-26T20:08:25.703015+0000","last_active":"2024-04-26T20:10:14.346963+0000","last_peered":"2024-04-26T20:10:14.346963+0000","last_clean":"2024-04-26T20:10:14.346963+0000","last_became_active":"2024-04-26T20:08:25.702699+0000","last_became_peered":"2024-04-26T20:08:25.702699+0000","last_unstale":"2024-04-26T20:10:14.346963+0000","last_undegraded":"2024-04-26T20:10:14.346963+0000","last_fullsized":"2024-04-26T20:10:14.346963+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:29:17.082803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.769010+0000","last_change":"2024-04-26T20:08:15.624092+0000","last_active":"2024-04-26T20:10:13.769010+0000","last_peered":"2024-04-26T20:10:13.769010+0000","last_clean":"2024-04-26T20:10:13.769010+0000","last_became_active":"2024-04-26T20:08:15.623540+0000","last_became_peered":"2024-04-26T20:08:15.623540+0000","last_unstale":"2024-04-26T20:10:13.769010+0000","last_undegraded":"2024-04-26T20:10:13.769010+0000","last_fullsized":"2024-04-26T20:10:13.769010+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:06:17.162854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766456+0000","last_change":"2024-04-26T20:09:13.313449+0000","last_active":"2024-04-26T20:10:13.766456+0000","last_peered":"2024-04-26T20:10:13.766456+0000","last_clean":"2024-04-26T20:10:13.766456+0000","last_became_active":"2024-04-26T20:09:13.313124+0000","last_became_peered":"2024-04-26T20:09:13.313124+0000","last_unstale":"2024-04-26T20:10:13.766456+0000","last_undegraded":"2024-04-26T20:10:13.766456+0000","last_fullsized":"2024-04-26T20:10:13.766456+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:10.447987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763999+0000","last_change":"2024-04-26T20:08:25.684973+0000","last_active":"2024-04-26T20:10:13.763999+0000","last_peered":"2024-04-26T20:10:13.763999+0000","last_clean":"2024-04-26T20:10:13.763999+0000","last_became_active":"2024-04-26T20:08:25.684636+0000","last_became_peered":"2024-04-26T20:08:25.684636+0000","last_unstale":"2024-04-26T20:10:13.763999+0000","last_undegraded":"2024-04-26T20:10:13.763999+0000","last_fullsized":"2024-04-26T20:10:13.763999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:47:17.140765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":158,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:00.997100+0000","last_change":"2024-04-26T20:10:13.934713+0000","last_active":"2024-04-26T20:11:00.997100+0000","last_peered":"2024-04-26T20:11:00.997100+0000","last_clean":"2024-04-26T20:11:00.997100+0000","last_became_active":"2024-04-26T20:10:13.930796+0000","last_became_peered":"2024-04-26T20:10:13.930796+0000","last_unstale":"2024-04-26T20:11:00.997100+0000","last_undegraded":"2024-04-26T20:11:00.997100+0000","last_fullsized":"2024-04-26T20:11:00.997100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:42:28.016674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768580+0000","last_change":"2024-04-26T20:08:17.660345+0000","last_active":"2024-04-26T20:10:13.768580+0000","last_peered":"2024-04-26T20:10:13.768580+0000","last_clean":"2024-04-26T20:10:13.768580+0000","last_became_active":"2024-04-26T20:08:17.659966+0000","last_became_peered":"2024-04-26T20:08:17.659966+0000","last_unstale":"2024-04-26T20:10:13.768580+0000","last_undegraded":"2024-04-26T20:10:13.768580+0000","last_fullsized":"2024-04-26T20:10:13.768580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:48.762984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763965+0000","last_change":"2024-04-26T20:08:25.698002+0000","last_active":"2024-04-26T20:10:13.763965+0000","last_peered":"2024-04-26T20:10:13.763965+0000","last_clean":"2024-04-26T20:10:13.763965+0000","last_became_active":"2024-04-26T20:08:25.697588+0000","last_became_peered":"2024-04-26T20:08:25.697588+0000","last_unstale":"2024-04-26T20:10:13.763965+0000","last_undegraded":"2024-04-26T20:10:13.763965+0000","last_fullsized":"2024-04-26T20:10:13.763965+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:59.771941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"94'3","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763425+0000","last_change":"2024-04-26T20:08:15.633759+0000","last_active":"2024-04-26T20:10:13.763425+0000","last_peered":"2024-04-26T20:10:13.763425+0000","last_clean":"2024-04-26T20:10:13.763425+0000","last_became_active":"2024-04-26T20:08:15.633354+0000","last_became_peered":"2024-04-26T20:08:15.633354+0000","last_unstale":"2024-04-26T20:10:13.763425+0000","last_undegraded":"2024-04-26T20:10:13.763425+0000","last_fullsized":"2024-04-26T20:10:13.763425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:33.394418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.769795+0000","last_change":"2024-04-26T20:09:13.319471+0000","last_active":"2024-04-26T20:10:13.769795+0000","last_peered":"2024-04-26T20:10:13.769795+0000","last_clean":"2024-04-26T20:10:13.769795+0000","last_became_active":"2024-04-26T20:09:13.318292+0000","last_became_peered":"2024-04-26T20:09:13.318292+0000","last_unstale":"2024-04-26T20:10:13.769795+0000","last_undegraded":"2024-04-26T20:10:13.769795+0000","last_fullsized":"2024-04-26T20:10:13.769795+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:41:36.278298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765836+0000","last_change":"2024-04-26T20:08:25.713338+0000","last_active":"2024-04-26T20:10:13.765836+0000","last_peered":"2024-04-26T20:10:13.765836+0000","last_clean":"2024-04-26T20:10:13.765836+0000","last_became_active":"2024-04-26T20:08:25.712905+0000","last_became_peered":"2024-04-26T20:08:25.712905+0000","last_unstale":"2024-04-26T20:10:13.765836+0000","last_undegraded":"2024-04-26T20:10:13.765836+0000","last_fullsized":"2024-04-26T20:10:13.765836+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:54.811957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"94'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767348+0000","last_change":"2024-04-26T20:08:15.633813+0000","last_active":"2024-04-26T20:10:13.767348+0000","last_peered":"2024-04-26T20:10:13.767348+0000","last_clean":"2024-04-26T20:10:13.767348+0000","last_became_active":"2024-04-26T20:08:15.633448+0000","last_became_peered":"2024-04-26T20:08:15.633448+0000","last_unstale":"2024-04-26T20:10:13.767348+0000","last_undegraded":"2024-04-26T20:10:13.767348+0000","last_fullsized":"2024-04-26T20:10:13.767348+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:52.909106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764347+0000","last_change":"2024-04-26T20:08:17.659606+0000","last_active":"2024-04-26T20:10:13.764347+0000","last_peered":"2024-04-26T20:10:13.764347+0000","last_clean":"2024-04-26T20:10:13.764347+0000","last_became_active":"2024-04-26T20:08:17.658948+0000","last_became_peered":"2024-04-26T20:08:17.658948+0000","last_unstale":"2024-04-26T20:10:13.764347+0000","last_undegraded":"2024-04-26T20:10:13.764347+0000","last_fullsized":"2024-04-26T20:10:13.764347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:03.616172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350555+0000","last_change":"2024-04-26T20:08:25.700149+0000","last_active":"2024-04-26T20:10:14.350555+0000","last_peered":"2024-04-26T20:10:14.350555+0000","last_clean":"2024-04-26T20:10:14.350555+0000","last_became_active":"2024-04-26T20:08:25.699693+0000","last_became_peered":"2024-04-26T20:08:25.699693+0000","last_unstale":"2024-04-26T20:10:14.350555+0000","last_undegraded":"2024-04-26T20:10:14.350555+0000","last_fullsized":"2024-04-26T20:10:14.350555+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:53.029443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"94'1","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765834+0000","last_change":"2024-04-26T20:08:15.626386+0000","last_active":"2024-04-26T20:10:13.765834+0000","last_peered":"2024-04-26T20:10:13.765834+0000","last_clean":"2024-04-26T20:10:13.765834+0000","last_became_active":"2024-04-26T20:08:15.626071+0000","last_became_peered":"2024-04-26T20:08:15.626071+0000","last_unstale":"2024-04-26T20:10:13.765834+0000","last_undegraded":"2024-04-26T20:10:13.765834+0000","last_fullsized":"2024-04-26T20:10:13.765834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:49.478038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770021+0000","last_change":"2024-04-26T20:08:17.668616+0000","last_active":"2024-04-26T20:10:13.770021+0000","last_peered":"2024-04-26T20:10:13.770021+0000","last_clean":"2024-04-26T20:10:13.770021+0000","last_became_active":"2024-04-26T20:08:17.668210+0000","last_became_peered":"2024-04-26T20:08:17.668210+0000","last_unstale":"2024-04-26T20:10:13.770021+0000","last_undegraded":"2024-04-26T20:10:13.770021+0000","last_fullsized":"2024-04-26T20:10:13.770021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:42.219107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763958+0000","last_change":"2024-04-26T20:08:25.699132+0000","last_active":"2024-04-26T20:10:13.763958+0000","last_peered":"2024-04-26T20:10:13.763958+0000","last_clean":"2024-04-26T20:10:13.763958+0000","last_became_active":"2024-04-26T20:08:25.698675+0000","last_became_peered":"2024-04-26T20:08:25.698675+0000","last_unstale":"2024-04-26T20:10:13.763958+0000","last_undegraded":"2024-04-26T20:10:13.763958+0000","last_fullsized":"2024-04-26T20:10:13.763958+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:08:21.950765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'3","reported_seq":142,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:02.999423+0000","last_change":"2024-04-26T20:08:15.627167+0000","last_active":"2024-04-26T20:11:02.999423+0000","last_peered":"2024-04-26T20:11:02.999423+0000","last_clean":"2024-04-26T20:11:02.999423+0000","last_became_active":"2024-04-26T20:08:15.625928+0000","last_became_peered":"2024-04-26T20:08:15.625928+0000","last_unstale":"2024-04-26T20:11:02.999423+0000","last_undegraded":"2024-04-26T20:11:02.999423+0000","last_fullsized":"2024-04-26T20:11:02.999423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:17:50.361864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930305+0000","last_change":"2024-04-26T20:08:17.652724+0000","last_active":"2024-04-26T20:10:13.930305+0000","last_peered":"2024-04-26T20:10:13.930305+0000","last_clean":"2024-04-26T20:10:13.930305+0000","last_became_active":"2024-04-26T20:08:17.652342+0000","last_became_peered":"2024-04-26T20:08:17.652342+0000","last_unstale":"2024-04-26T20:10:13.930305+0000","last_undegraded":"2024-04-26T20:10:13.930305+0000","last_fullsized":"2024-04-26T20:10:13.930305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:04:37.633017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.769762+0000","last_change":"2024-04-26T20:08:25.703811+0000","last_active":"2024-04-26T20:10:13.769762+0000","last_peered":"2024-04-26T20:10:13.769762+0000","last_clean":"2024-04-26T20:10:13.769762+0000","last_became_active":"2024-04-26T20:08:25.703247+0000","last_became_peered":"2024-04-26T20:08:25.703247+0000","last_unstale":"2024-04-26T20:10:13.769762+0000","last_undegraded":"2024-04-26T20:10:13.769762+0000","last_fullsized":"2024-04-26T20:10:13.769762+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:19.605775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"94'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767779+0000","last_change":"2024-04-26T20:08:15.628275+0000","last_active":"2024-04-26T20:10:13.767779+0000","last_peered":"2024-04-26T20:10:13.767779+0000","last_clean":"2024-04-26T20:10:13.767779+0000","last_became_active":"2024-04-26T20:08:15.627115+0000","last_became_peered":"2024-04-26T20:08:15.627115+0000","last_unstale":"2024-04-26T20:10:13.767779+0000","last_undegraded":"2024-04-26T20:10:13.767779+0000","last_fullsized":"2024-04-26T20:10:13.767779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:48.799646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.351023+0000","last_change":"2024-04-26T20:08:17.651666+0000","last_active":"2024-04-26T20:10:14.351023+0000","last_peered":"2024-04-26T20:10:14.351023+0000","last_clean":"2024-04-26T20:10:14.351023+0000","last_became_active":"2024-04-26T20:08:17.644762+0000","last_became_peered":"2024-04-26T20:08:17.644762+0000","last_unstale":"2024-04-26T20:10:14.351023+0000","last_undegraded":"2024-04-26T20:10:14.351023+0000","last_fullsized":"2024-04-26T20:10:14.351023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:19.070044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347175+0000","last_change":"2024-04-26T20:09:13.314326+0000","last_active":"2024-04-26T20:10:14.347175+0000","last_peered":"2024-04-26T20:10:14.347175+0000","last_clean":"2024-04-26T20:10:14.347175+0000","last_became_active":"2024-04-26T20:09:13.313823+0000","last_became_peered":"2024-04-26T20:09:13.313823+0000","last_unstale":"2024-04-26T20:10:14.347175+0000","last_undegraded":"2024-04-26T20:10:14.347175+0000","last_fullsized":"2024-04-26T20:10:14.347175+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:12:55.398462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'4","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766361+0000","last_change":"2024-04-26T20:08:16.123015+0000","last_active":"2024-04-26T20:10:13.766361+0000","last_peered":"2024-04-26T20:10:13.766361+0000","last_clean":"2024-04-26T20:10:13.766361+0000","last_became_active":"2024-04-26T20:08:16.122594+0000","last_became_peered":"2024-04-26T20:08:16.122594+0000","last_unstale":"2024-04-26T20:10:13.766361+0000","last_undegraded":"2024-04-26T20:10:13.766361+0000","last_fullsized":"2024-04-26T20:10:13.766361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:54:11.846029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930053+0000","last_change":"2024-04-26T20:08:17.640882+0000","last_active":"2024-04-26T20:10:13.930053+0000","last_peered":"2024-04-26T20:10:13.930053+0000","last_clean":"2024-04-26T20:10:13.930053+0000","last_became_active":"2024-04-26T20:08:17.640399+0000","last_became_peered":"2024-04-26T20:08:17.640399+0000","last_unstale":"2024-04-26T20:10:13.930053+0000","last_undegraded":"2024-04-26T20:10:13.930053+0000","last_fullsized":"2024-04-26T20:10:13.930053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:05:57.554751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770025+0000","last_change":"2024-04-26T20:08:25.713003+0000","last_active":"2024-04-26T20:10:13.770025+0000","last_peered":"2024-04-26T20:10:13.770025+0000","last_clean":"2024-04-26T20:10:13.770025+0000","last_became_active":"2024-04-26T20:08:25.711116+0000","last_became_peered":"2024-04-26T20:08:25.711116+0000","last_unstale":"2024-04-26T20:10:13.770025+0000","last_undegraded":"2024-04-26T20:10:13.770025+0000","last_fullsized":"2024-04-26T20:10:13.770025+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:45:50.627388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'34","reported_seq":282,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:04.000653+0000","last_change":"2024-04-26T20:08:15.629344+0000","last_active":"2024-04-26T20:11:04.000653+0000","last_peered":"2024-04-26T20:11:04.000653+0000","last_clean":"2024-04-26T20:11:04.000653+0000","last_became_active":"2024-04-26T20:08:15.628836+0000","last_became_peered":"2024-04-26T20:08:15.628836+0000","last_unstale":"2024-04-26T20:11:04.000653+0000","last_undegraded":"2024-04-26T20:11:04.000653+0000","last_fullsized":"2024-04-26T20:11:04.000653+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:05:27.011109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57111,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":209,"num_read_kb":160,"num_write":35,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767737+0000","last_change":"2024-04-26T20:08:17.641469+0000","last_active":"2024-04-26T20:10:13.767737+0000","last_peered":"2024-04-26T20:10:13.767737+0000","last_clean":"2024-04-26T20:10:13.767737+0000","last_became_active":"2024-04-26T20:08:17.641147+0000","last_became_peered":"2024-04-26T20:08:17.641147+0000","last_unstale":"2024-04-26T20:10:13.767737+0000","last_undegraded":"2024-04-26T20:10:13.767737+0000","last_fullsized":"2024-04-26T20:10:13.767737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:36:02.885095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766776+0000","last_change":"2024-04-26T20:08:25.708459+0000","last_active":"2024-04-26T20:10:13.766776+0000","last_peered":"2024-04-26T20:10:13.766776+0000","last_clean":"2024-04-26T20:10:13.766776+0000","last_became_active":"2024-04-26T20:08:25.708099+0000","last_became_peered":"2024-04-26T20:08:25.708099+0000","last_unstale":"2024-04-26T20:10:13.766776+0000","last_undegraded":"2024-04-26T20:10:13.766776+0000","last_fullsized":"2024-04-26T20:10:13.766776+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:37.423731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"94'6","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764032+0000","last_change":"2024-04-26T20:08:15.627399+0000","last_active":"2024-04-26T20:10:13.764032+0000","last_peered":"2024-04-26T20:10:13.764032+0000","last_clean":"2024-04-26T20:10:13.764032+0000","last_became_active":"2024-04-26T20:08:15.627018+0000","last_became_peered":"2024-04-26T20:08:15.627018+0000","last_unstale":"2024-04-26T20:10:13.764032+0000","last_undegraded":"2024-04-26T20:10:13.764032+0000","last_fullsized":"2024-04-26T20:10:13.764032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:13.491035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767541+0000","last_change":"2024-04-26T20:08:17.669883+0000","last_active":"2024-04-26T20:10:13.767541+0000","last_peered":"2024-04-26T20:10:13.767541+0000","last_clean":"2024-04-26T20:10:13.767541+0000","last_became_active":"2024-04-26T20:08:17.669484+0000","last_became_peered":"2024-04-26T20:08:17.669484+0000","last_unstale":"2024-04-26T20:10:13.767541+0000","last_undegraded":"2024-04-26T20:10:13.767541+0000","last_fullsized":"2024-04-26T20:10:13.767541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:53:28.820771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765415+0000","last_change":"2024-04-26T20:08:25.694352+0000","last_active":"2024-04-26T20:10:13.765415+0000","last_peered":"2024-04-26T20:10:13.765415+0000","last_clean":"2024-04-26T20:10:13.765415+0000","last_became_active":"2024-04-26T20:08:25.694022+0000","last_became_peered":"2024-04-26T20:08:25.694022+0000","last_unstale":"2024-04-26T20:10:13.765415+0000","last_undegraded":"2024-04-26T20:10:13.765415+0000","last_fullsized":"2024-04-26T20:10:13.765415+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:23.830964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"94'3","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766290+0000","last_change":"2024-04-26T20:08:15.625816+0000","last_active":"2024-04-26T20:10:13.766290+0000","last_peered":"2024-04-26T20:10:13.766290+0000","last_clean":"2024-04-26T20:10:13.766290+0000","last_became_active":"2024-04-26T20:08:15.625390+0000","last_became_peered":"2024-04-26T20:08:15.625390+0000","last_unstale":"2024-04-26T20:10:13.766290+0000","last_undegraded":"2024-04-26T20:10:13.766290+0000","last_fullsized":"2024-04-26T20:10:13.766290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:08:57.864795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767621+0000","last_change":"2024-04-26T20:08:17.653016+0000","last_active":"2024-04-26T20:10:13.767621+0000","last_peered":"2024-04-26T20:10:13.767621+0000","last_clean":"2024-04-26T20:10:13.767621+0000","last_became_active":"2024-04-26T20:08:17.652403+0000","last_became_peered":"2024-04-26T20:08:17.652403+0000","last_unstale":"2024-04-26T20:10:13.767621+0000","last_undegraded":"2024-04-26T20:10:13.767621+0000","last_fullsized":"2024-04-26T20:10:13.767621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:02.580052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770803+0000","last_change":"2024-04-26T20:08:25.717469+0000","last_active":"2024-04-26T20:10:13.770803+0000","last_peered":"2024-04-26T20:10:13.770803+0000","last_clean":"2024-04-26T20:10:13.770803+0000","last_became_active":"2024-04-26T20:08:25.716701+0000","last_became_peered":"2024-04-26T20:08:25.716701+0000","last_unstale":"2024-04-26T20:10:13.770803+0000","last_undegraded":"2024-04-26T20:10:13.770803+0000","last_fullsized":"2024-04-26T20:10:13.770803+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:11:10.991196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'3","reported_seq":143,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:01.755779+0000","last_change":"2024-04-26T20:08:15.626268+0000","last_active":"2024-04-26T20:11:01.755779+0000","last_peered":"2024-04-26T20:11:01.755779+0000","last_clean":"2024-04-26T20:11:01.755779+0000","last_became_active":"2024-04-26T20:08:15.625839+0000","last_became_peered":"2024-04-26T20:08:15.625839+0000","last_unstale":"2024-04-26T20:11:01.755779+0000","last_undegraded":"2024-04-26T20:11:01.755779+0000","last_fullsized":"2024-04-26T20:11:01.755779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:54.537335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767457+0000","last_change":"2024-04-26T20:08:17.644651+0000","last_active":"2024-04-26T20:10:13.767457+0000","last_peered":"2024-04-26T20:10:13.767457+0000","last_clean":"2024-04-26T20:10:13.767457+0000","last_became_active":"2024-04-26T20:08:17.644357+0000","last_became_peered":"2024-04-26T20:08:17.644357+0000","last_unstale":"2024-04-26T20:10:13.767457+0000","last_undegraded":"2024-04-26T20:10:13.767457+0000","last_fullsized":"2024-04-26T20:10:13.767457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:36.344501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768962+0000","last_change":"2024-04-26T20:08:25.710016+0000","last_active":"2024-04-26T20:10:13.768962+0000","last_peered":"2024-04-26T20:10:13.768962+0000","last_clean":"2024-04-26T20:10:13.768962+0000","last_became_active":"2024-04-26T20:08:25.709647+0000","last_became_peered":"2024-04-26T20:08:25.709647+0000","last_unstale":"2024-04-26T20:10:13.768962+0000","last_undegraded":"2024-04-26T20:10:13.768962+0000","last_fullsized":"2024-04-26T20:10:13.768962+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:22.434116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"94'15","reported_seq":55,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768158+0000","last_change":"2024-04-26T20:08:16.128347+0000","last_active":"2024-04-26T20:10:13.768158+0000","last_peered":"2024-04-26T20:10:13.768158+0000","last_clean":"2024-04-26T20:10:13.768158+0000","last_became_active":"2024-04-26T20:08:16.127795+0000","last_became_peered":"2024-04-26T20:08:16.127795+0000","last_unstale":"2024-04-26T20:10:13.768158+0000","last_undegraded":"2024-04-26T20:10:13.768158+0000","last_fullsized":"2024-04-26T20:10:13.768158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:21.217202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770176+0000","last_change":"2024-04-26T20:09:13.314298+0000","last_active":"2024-04-26T20:10:13.770176+0000","last_peered":"2024-04-26T20:10:13.770176+0000","last_clean":"2024-04-26T20:10:13.770176+0000","last_became_active":"2024-04-26T20:09:13.313732+0000","last_became_peered":"2024-04-26T20:09:13.313732+0000","last_unstale":"2024-04-26T20:10:13.770176+0000","last_undegraded":"2024-04-26T20:10:13.770176+0000","last_fullsized":"2024-04-26T20:10:13.770176+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:59:41.782682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765794+0000","last_change":"2024-04-26T20:08:25.700504+0000","last_active":"2024-04-26T20:10:13.765794+0000","last_peered":"2024-04-26T20:10:13.765794+0000","last_clean":"2024-04-26T20:10:13.765794+0000","last_became_active":"2024-04-26T20:08:25.700171+0000","last_became_peered":"2024-04-26T20:08:25.700171+0000","last_unstale":"2024-04-26T20:10:13.765794+0000","last_undegraded":"2024-04-26T20:10:13.765794+0000","last_fullsized":"2024-04-26T20:10:13.765794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:04.442343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'2","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768449+0000","last_change":"2024-04-26T20:08:15.625738+0000","last_active":"2024-04-26T20:10:13.768449+0000","last_peered":"2024-04-26T20:10:13.768449+0000","last_clean":"2024-04-26T20:10:13.768449+0000","last_became_active":"2024-04-26T20:08:15.625381+0000","last_became_peered":"2024-04-26T20:08:15.625381+0000","last_unstale":"2024-04-26T20:10:13.768449+0000","last_undegraded":"2024-04-26T20:10:13.768449+0000","last_fullsized":"2024-04-26T20:10:13.768449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:52.378477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765694+0000","last_change":"2024-04-26T20:08:17.646682+0000","last_active":"2024-04-26T20:10:13.765694+0000","last_peered":"2024-04-26T20:10:13.765694+0000","last_clean":"2024-04-26T20:10:13.765694+0000","last_became_active":"2024-04-26T20:08:17.646362+0000","last_became_peered":"2024-04-26T20:08:17.646362+0000","last_unstale":"2024-04-26T20:10:13.765694+0000","last_undegraded":"2024-04-26T20:10:13.765694+0000","last_fullsized":"2024-04-26T20:10:13.765694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:52.955883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768718+0000","last_change":"2024-04-26T20:08:25.704580+0000","last_active":"2024-04-26T20:10:13.768718+0000","last_peered":"2024-04-26T20:10:13.768718+0000","last_clean":"2024-04-26T20:10:13.768718+0000","last_became_active":"2024-04-26T20:08:25.704083+0000","last_became_peered":"2024-04-26T20:08:25.704083+0000","last_unstale":"2024-04-26T20:10:13.768718+0000","last_undegraded":"2024-04-26T20:10:13.768718+0000","last_fullsized":"2024-04-26T20:10:13.768718+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:59:33.409501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770270+0000","last_change":"2024-04-26T20:08:16.131048+0000","last_active":"2024-04-26T20:10:13.770270+0000","last_peered":"2024-04-26T20:10:13.770270+0000","last_clean":"2024-04-26T20:10:13.770270+0000","last_became_active":"2024-04-26T20:08:16.129700+0000","last_became_peered":"2024-04-26T20:08:16.129700+0000","last_unstale":"2024-04-26T20:10:13.770270+0000","last_undegraded":"2024-04-26T20:10:13.770270+0000","last_fullsized":"2024-04-26T20:10:13.770270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:59:34.814697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763546+0000","last_change":"2024-04-26T20:08:17.669408+0000","last_active":"2024-04-26T20:10:13.763546+0000","last_peered":"2024-04-26T20:10:13.763546+0000","last_clean":"2024-04-26T20:10:13.763546+0000","last_became_active":"2024-04-26T20:08:17.669102+0000","last_became_peered":"2024-04-26T20:08:17.669102+0000","last_unstale":"2024-04-26T20:10:13.763546+0000","last_undegraded":"2024-04-26T20:10:13.763546+0000","last_fullsized":"2024-04-26T20:10:13.763546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:55:29.812387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768743+0000","last_change":"2024-04-26T20:08:25.692910+0000","last_active":"2024-04-26T20:10:13.768743+0000","last_peered":"2024-04-26T20:10:13.768743+0000","last_clean":"2024-04-26T20:10:13.768743+0000","last_became_active":"2024-04-26T20:08:25.692292+0000","last_became_peered":"2024-04-26T20:08:25.692292+0000","last_unstale":"2024-04-26T20:10:13.768743+0000","last_undegraded":"2024-04-26T20:10:13.768743+0000","last_fullsized":"2024-04-26T20:10:13.768743+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:42:19.483756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'1","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768002+0000","last_change":"2024-04-26T20:08:15.633948+0000","last_active":"2024-04-26T20:10:13.768002+0000","last_peered":"2024-04-26T20:10:13.768002+0000","last_clean":"2024-04-26T20:10:13.768002+0000","last_became_active":"2024-04-26T20:08:15.633531+0000","last_became_peered":"2024-04-26T20:08:15.633531+0000","last_unstale":"2024-04-26T20:10:13.768002+0000","last_undegraded":"2024-04-26T20:10:13.768002+0000","last_fullsized":"2024-04-26T20:10:13.768002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:13:15.587856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763403+0000","last_change":"2024-04-26T20:08:17.663674+0000","last_active":"2024-04-26T20:10:13.763403+0000","last_peered":"2024-04-26T20:10:13.763403+0000","last_clean":"2024-04-26T20:10:13.763403+0000","last_became_active":"2024-04-26T20:08:17.663309+0000","last_became_peered":"2024-04-26T20:08:17.663309+0000","last_unstale":"2024-04-26T20:10:13.763403+0000","last_undegraded":"2024-04-26T20:10:13.763403+0000","last_fullsized":"2024-04-26T20:10:13.763403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:20.661026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.769298+0000","last_change":"2024-04-26T20:08:25.698807+0000","last_active":"2024-04-26T20:10:13.769298+0000","last_peered":"2024-04-26T20:10:13.769298+0000","last_clean":"2024-04-26T20:10:13.769298+0000","last_became_active":"2024-04-26T20:08:25.697686+0000","last_became_peered":"2024-04-26T20:08:25.697686+0000","last_unstale":"2024-04-26T20:10:13.769298+0000","last_undegraded":"2024-04-26T20:10:13.769298+0000","last_fullsized":"2024-04-26T20:10:13.769298+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:59:37.099531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766429+0000","last_change":"2024-04-26T20:08:17.645055+0000","last_active":"2024-04-26T20:10:13.766429+0000","last_peered":"2024-04-26T20:10:13.766429+0000","last_clean":"2024-04-26T20:10:13.766429+0000","last_became_active":"2024-04-26T20:08:17.644715+0000","last_became_peered":"2024-04-26T20:08:17.644715+0000","last_unstale":"2024-04-26T20:10:13.766429+0000","last_undegraded":"2024-04-26T20:10:13.766429+0000","last_fullsized":"2024-04-26T20:10:13.766429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:12.508925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'4","reported_seq":146,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:05.764100+0000","last_change":"2024-04-26T20:08:16.123710+0000","last_active":"2024-04-26T20:11:05.764100+0000","last_peered":"2024-04-26T20:11:05.764100+0000","last_clean":"2024-04-26T20:11:05.764100+0000","last_became_active":"2024-04-26T20:08:16.123046+0000","last_became_peered":"2024-04-26T20:08:16.123046+0000","last_unstale":"2024-04-26T20:11:05.764100+0000","last_undegraded":"2024-04-26T20:11:05.764100+0000","last_fullsized":"2024-04-26T20:11:05.764100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:42.389448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"94'1","reported_seq":27,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767798+0000","last_change":"2024-04-26T20:08:25.698716+0000","last_active":"2024-04-26T20:10:13.767798+0000","last_peered":"2024-04-26T20:10:13.767798+0000","last_clean":"2024-04-26T20:10:13.767798+0000","last_became_active":"2024-04-26T20:08:25.698418+0000","last_became_peered":"2024-04-26T20:08:25.698418+0000","last_unstale":"2024-04-26T20:10:13.767798+0000","last_undegraded":"2024-04-26T20:10:13.767798+0000","last_fullsized":"2024-04-26T20:10:13.767798+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:56.290241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764439+0000","last_change":"2024-04-26T20:08:17.656393+0000","last_active":"2024-04-26T20:10:13.764439+0000","last_peered":"2024-04-26T20:10:13.764439+0000","last_clean":"2024-04-26T20:10:13.764439+0000","last_became_active":"2024-04-26T20:08:17.655914+0000","last_became_peered":"2024-04-26T20:08:17.655914+0000","last_unstale":"2024-04-26T20:10:13.764439+0000","last_undegraded":"2024-04-26T20:10:13.764439+0000","last_fullsized":"2024-04-26T20:10:13.764439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:38:41.459519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"94'3","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347109+0000","last_change":"2024-04-26T20:08:15.631983+0000","last_active":"2024-04-26T20:10:14.347109+0000","last_peered":"2024-04-26T20:10:14.347109+0000","last_clean":"2024-04-26T20:10:14.347109+0000","last_became_active":"2024-04-26T20:08:15.631625+0000","last_became_peered":"2024-04-26T20:08:15.631625+0000","last_unstale":"2024-04-26T20:10:14.347109+0000","last_undegraded":"2024-04-26T20:10:14.347109+0000","last_fullsized":"2024-04-26T20:10:14.347109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:49.264149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":362,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767288+0000","last_change":"2024-04-26T20:06:47.857115+0000","last_active":"2024-04-26T20:10:13.767288+0000","last_peered":"2024-04-26T20:10:13.767288+0000","last_clean":"2024-04-26T20:10:13.767288+0000","last_became_active":"2024-04-26T20:06:47.856652+0000","last_became_peered":"2024-04-26T20:06:47.856652+0000","last_unstale":"2024-04-26T20:10:13.767288+0000","last_undegraded":"2024-04-26T20:10:13.767288+0000","last_fullsized":"2024-04-26T20:10:13.767288+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:02:53.903427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:02:53.903427+0000","last_clean_scrub_stamp":"2024-04-26T20:02:53.903427+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:35:14.359328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768095+0000","last_change":"2024-04-26T20:09:13.283142+0000","last_active":"2024-04-26T20:10:13.768095+0000","last_peered":"2024-04-26T20:10:13.768095+0000","last_clean":"2024-04-26T20:10:13.768095+0000","last_became_active":"2024-04-26T20:09:13.282806+0000","last_became_peered":"2024-04-26T20:09:13.282806+0000","last_unstale":"2024-04-26T20:10:13.768095+0000","last_undegraded":"2024-04-26T20:10:13.768095+0000","last_fullsized":"2024-04-26T20:10:13.768095+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:39.595299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767428+0000","last_change":"2024-04-26T20:08:17.654402+0000","last_active":"2024-04-26T20:10:13.767428+0000","last_peered":"2024-04-26T20:10:13.767428+0000","last_clean":"2024-04-26T20:10:13.767428+0000","last_became_active":"2024-04-26T20:08:17.650047+0000","last_became_peered":"2024-04-26T20:08:17.650047+0000","last_unstale":"2024-04-26T20:10:13.767428+0000","last_undegraded":"2024-04-26T20:10:13.767428+0000","last_fullsized":"2024-04-26T20:10:13.767428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:40.225984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"94'3","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767490+0000","last_change":"2024-04-26T20:08:15.626610+0000","last_active":"2024-04-26T20:10:13.767490+0000","last_peered":"2024-04-26T20:10:13.767490+0000","last_clean":"2024-04-26T20:10:13.767490+0000","last_became_active":"2024-04-26T20:08:15.626162+0000","last_became_peered":"2024-04-26T20:08:15.626162+0000","last_unstale":"2024-04-26T20:10:13.767490+0000","last_undegraded":"2024-04-26T20:10:13.767490+0000","last_fullsized":"2024-04-26T20:10:13.767490+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:20.270714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765912+0000","last_change":"2024-04-26T20:08:25.697123+0000","last_active":"2024-04-26T20:10:13.765912+0000","last_peered":"2024-04-26T20:10:13.765912+0000","last_clean":"2024-04-26T20:10:13.765912+0000","last_became_active":"2024-04-26T20:08:25.696773+0000","last_became_peered":"2024-04-26T20:08:25.696773+0000","last_unstale":"2024-04-26T20:10:13.765912+0000","last_undegraded":"2024-04-26T20:10:13.765912+0000","last_fullsized":"2024-04-26T20:10:13.765912+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:22:56.642427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"94'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767840+0000","last_change":"2024-04-26T20:08:16.128206+0000","last_active":"2024-04-26T20:10:13.767840+0000","last_peered":"2024-04-26T20:10:13.767840+0000","last_clean":"2024-04-26T20:10:13.767840+0000","last_became_active":"2024-04-26T20:08:16.127703+0000","last_became_peered":"2024-04-26T20:08:16.127703+0000","last_unstale":"2024-04-26T20:10:13.767840+0000","last_undegraded":"2024-04-26T20:10:13.767840+0000","last_fullsized":"2024-04-26T20:10:13.767840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:39.852292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768512+0000","last_change":"2024-04-26T20:08:17.652810+0000","last_active":"2024-04-26T20:10:13.768512+0000","last_peered":"2024-04-26T20:10:13.768512+0000","last_clean":"2024-04-26T20:10:13.768512+0000","last_became_active":"2024-04-26T20:08:17.652486+0000","last_became_peered":"2024-04-26T20:08:17.652486+0000","last_unstale":"2024-04-26T20:10:13.768512+0000","last_undegraded":"2024-04-26T20:10:13.768512+0000","last_fullsized":"2024-04-26T20:10:13.768512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:43.382814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764175+0000","last_change":"2024-04-26T20:08:25.701678+0000","last_active":"2024-04-26T20:10:13.764175+0000","last_peered":"2024-04-26T20:10:13.764175+0000","last_clean":"2024-04-26T20:10:13.764175+0000","last_became_active":"2024-04-26T20:08:25.701283+0000","last_became_peered":"2024-04-26T20:08:25.701283+0000","last_unstale":"2024-04-26T20:10:13.764175+0000","last_undegraded":"2024-04-26T20:10:13.764175+0000","last_fullsized":"2024-04-26T20:10:13.764175+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:35:26.774248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"96'11","reported_seq":50,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:27.763015+0000","last_change":"2024-04-26T20:08:15.634215+0000","last_active":"2024-04-26T20:10:27.763015+0000","last_peered":"2024-04-26T20:10:27.763015+0000","last_clean":"2024-04-26T20:10:27.763015+0000","last_became_active":"2024-04-26T20:08:15.633669+0000","last_became_peered":"2024-04-26T20:08:15.633669+0000","last_unstale":"2024-04-26T20:10:27.763015+0000","last_undegraded":"2024-04-26T20:10:27.763015+0000","last_fullsized":"2024-04-26T20:10:27.763015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:06.178974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765223+0000","last_change":"2024-04-26T20:08:17.666557+0000","last_active":"2024-04-26T20:10:13.765223+0000","last_peered":"2024-04-26T20:10:13.765223+0000","last_clean":"2024-04-26T20:10:13.765223+0000","last_became_active":"2024-04-26T20:08:17.665552+0000","last_became_peered":"2024-04-26T20:08:17.665552+0000","last_unstale":"2024-04-26T20:10:13.765223+0000","last_undegraded":"2024-04-26T20:10:13.765223+0000","last_fullsized":"2024-04-26T20:10:13.765223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:11.993087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763576+0000","last_change":"2024-04-26T20:08:25.684541+0000","last_active":"2024-04-26T20:10:13.763576+0000","last_peered":"2024-04-26T20:10:13.763576+0000","last_clean":"2024-04-26T20:10:13.763576+0000","last_became_active":"2024-04-26T20:08:25.684159+0000","last_became_peered":"2024-04-26T20:08:25.684159+0000","last_unstale":"2024-04-26T20:10:13.763576+0000","last_undegraded":"2024-04-26T20:10:13.763576+0000","last_fullsized":"2024-04-26T20:10:13.763576+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:31:45.405767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"94'4","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347923+0000","last_change":"2024-04-26T20:08:16.002624+0000","last_active":"2024-04-26T20:10:14.347923+0000","last_peered":"2024-04-26T20:10:14.347923+0000","last_clean":"2024-04-26T20:10:14.347923+0000","last_became_active":"2024-04-26T20:08:16.002246+0000","last_became_peered":"2024-04-26T20:08:16.002246+0000","last_unstale":"2024-04-26T20:10:14.347923+0000","last_undegraded":"2024-04-26T20:10:14.347923+0000","last_fullsized":"2024-04-26T20:10:14.347923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:59.393586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350768+0000","last_change":"2024-04-26T20:09:13.314961+0000","last_active":"2024-04-26T20:10:14.350768+0000","last_peered":"2024-04-26T20:10:14.350768+0000","last_clean":"2024-04-26T20:10:14.350768+0000","last_became_active":"2024-04-26T20:09:13.314533+0000","last_became_peered":"2024-04-26T20:09:13.314533+0000","last_unstale":"2024-04-26T20:10:14.350768+0000","last_undegraded":"2024-04-26T20:10:14.350768+0000","last_fullsized":"2024-04-26T20:10:14.350768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:29:27.518198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347800+0000","last_change":"2024-04-26T20:08:25.713891+0000","last_active":"2024-04-26T20:10:14.347800+0000","last_peered":"2024-04-26T20:10:14.347800+0000","last_clean":"2024-04-26T20:10:14.347800+0000","last_became_active":"2024-04-26T20:08:25.713480+0000","last_became_peered":"2024-04-26T20:08:25.713480+0000","last_unstale":"2024-04-26T20:10:14.347800+0000","last_undegraded":"2024-04-26T20:10:14.347800+0000","last_fullsized":"2024-04-26T20:10:14.347800+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:26.226811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"94'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.929950+0000","last_change":"2024-04-26T20:08:16.123332+0000","last_active":"2024-04-26T20:10:13.929950+0000","last_peered":"2024-04-26T20:10:13.929950+0000","last_clean":"2024-04-26T20:10:13.929950+0000","last_became_active":"2024-04-26T20:08:16.122944+0000","last_became_peered":"2024-04-26T20:08:16.122944+0000","last_unstale":"2024-04-26T20:10:13.929950+0000","last_undegraded":"2024-04-26T20:10:13.929950+0000","last_fullsized":"2024-04-26T20:10:13.929950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:37:23.369867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767503+0000","last_change":"2024-04-26T20:08:17.641693+0000","last_active":"2024-04-26T20:10:13.767503+0000","last_peered":"2024-04-26T20:10:13.767503+0000","last_clean":"2024-04-26T20:10:13.767503+0000","last_became_active":"2024-04-26T20:08:17.641334+0000","last_became_peered":"2024-04-26T20:08:17.641334+0000","last_unstale":"2024-04-26T20:10:13.767503+0000","last_undegraded":"2024-04-26T20:10:13.767503+0000","last_fullsized":"2024-04-26T20:10:13.767503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:05.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770610+0000","last_change":"2024-04-26T20:08:25.714554+0000","last_active":"2024-04-26T20:10:13.770610+0000","last_peered":"2024-04-26T20:10:13.770610+0000","last_clean":"2024-04-26T20:10:13.770610+0000","last_became_active":"2024-04-26T20:08:25.711975+0000","last_became_peered":"2024-04-26T20:08:25.711975+0000","last_unstale":"2024-04-26T20:10:13.770610+0000","last_undegraded":"2024-04-26T20:10:13.770610+0000","last_fullsized":"2024-04-26T20:10:13.770610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:20.061288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"94'3","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764134+0000","last_change":"2024-04-26T20:08:15.634841+0000","last_active":"2024-04-26T20:10:13.764134+0000","last_peered":"2024-04-26T20:10:13.764134+0000","last_clean":"2024-04-26T20:10:13.764134+0000","last_became_active":"2024-04-26T20:08:15.633138+0000","last_became_peered":"2024-04-26T20:08:15.633138+0000","last_unstale":"2024-04-26T20:10:13.764134+0000","last_undegraded":"2024-04-26T20:10:13.764134+0000","last_fullsized":"2024-04-26T20:10:13.764134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:19.364982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765341+0000","last_change":"2024-04-26T20:08:17.650201+0000","last_active":"2024-04-26T20:10:13.765341+0000","last_peered":"2024-04-26T20:10:13.765341+0000","last_clean":"2024-04-26T20:10:13.765341+0000","last_became_active":"2024-04-26T20:08:17.649819+0000","last_became_peered":"2024-04-26T20:08:17.649819+0000","last_unstale":"2024-04-26T20:10:13.765341+0000","last_undegraded":"2024-04-26T20:10:13.765341+0000","last_fullsized":"2024-04-26T20:10:13.765341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:01.772268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"94'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347627+0000","last_change":"2024-04-26T20:09:13.315540+0000","last_active":"2024-04-26T20:10:14.347627+0000","last_peered":"2024-04-26T20:10:14.347627+0000","last_clean":"2024-04-26T20:10:14.347627+0000","last_became_active":"2024-04-26T20:09:13.314599+0000","last_became_peered":"2024-04-26T20:09:13.314599+0000","last_unstale":"2024-04-26T20:10:14.347627+0000","last_undegraded":"2024-04-26T20:10:14.347627+0000","last_fullsized":"2024-04-26T20:10:14.347627+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:29.926505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'1","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763660+0000","last_change":"2024-04-26T20:08:15.625290+0000","last_active":"2024-04-26T20:10:13.763660+0000","last_peered":"2024-04-26T20:10:13.763660+0000","last_clean":"2024-04-26T20:10:13.763660+0000","last_became_active":"2024-04-26T20:08:15.624310+0000","last_became_peered":"2024-04-26T20:08:15.624310+0000","last_unstale":"2024-04-26T20:10:13.763660+0000","last_undegraded":"2024-04-26T20:10:13.763660+0000","last_fullsized":"2024-04-26T20:10:13.763660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:04:16.736552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764599+0000","last_change":"2024-04-26T20:08:17.665648+0000","last_active":"2024-04-26T20:10:13.764599+0000","last_peered":"2024-04-26T20:10:13.764599+0000","last_clean":"2024-04-26T20:10:13.764599+0000","last_became_active":"2024-04-26T20:08:17.665205+0000","last_became_peered":"2024-04-26T20:08:17.665205+0000","last_unstale":"2024-04-26T20:10:13.764599+0000","last_undegraded":"2024-04-26T20:10:13.764599+0000","last_fullsized":"2024-04-26T20:10:13.764599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:51.138181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347741+0000","last_change":"2024-04-26T20:08:25.708815+0000","last_active":"2024-04-26T20:10:14.347741+0000","last_peered":"2024-04-26T20:10:14.347741+0000","last_clean":"2024-04-26T20:10:14.347741+0000","last_became_active":"2024-04-26T20:08:25.708332+0000","last_became_peered":"2024-04-26T20:08:25.708332+0000","last_unstale":"2024-04-26T20:10:14.347741+0000","last_undegraded":"2024-04-26T20:10:14.347741+0000","last_fullsized":"2024-04-26T20:10:14.347741+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:23:19.378956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"94'4","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.773587+0000","last_change":"2024-04-26T20:10:13.773165+0000","last_active":"2024-04-26T20:10:13.773587+0000","last_peered":"2024-04-26T20:10:13.773587+0000","last_clean":"2024-04-26T20:10:13.773587+0000","last_became_active":"2024-04-26T20:10:13.768588+0000","last_became_peered":"2024-04-26T20:10:13.768588+0000","last_unstale":"2024-04-26T20:10:13.773587+0000","last_undegraded":"2024-04-26T20:10:13.773587+0000","last_fullsized":"2024-04-26T20:10:13.773587+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:15.035075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767991+0000","last_change":"2024-04-26T20:08:17.658650+0000","last_active":"2024-04-26T20:10:13.767991+0000","last_peered":"2024-04-26T20:10:13.767991+0000","last_clean":"2024-04-26T20:10:13.767991+0000","last_became_active":"2024-04-26T20:08:17.658325+0000","last_became_peered":"2024-04-26T20:08:17.658325+0000","last_unstale":"2024-04-26T20:10:13.767991+0000","last_undegraded":"2024-04-26T20:10:13.767991+0000","last_fullsized":"2024-04-26T20:10:13.767991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:50:27.167289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.929885+0000","last_change":"2024-04-26T20:08:25.681040+0000","last_active":"2024-04-26T20:10:13.929885+0000","last_peered":"2024-04-26T20:10:13.929885+0000","last_clean":"2024-04-26T20:10:13.929885+0000","last_became_active":"2024-04-26T20:08:25.680632+0000","last_became_peered":"2024-04-26T20:08:25.680632+0000","last_unstale":"2024-04-26T20:10:13.929885+0000","last_undegraded":"2024-04-26T20:10:13.929885+0000","last_fullsized":"2024-04-26T20:10:13.929885+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:16.017414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'3","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764076+0000","last_change":"2024-04-26T20:08:16.123274+0000","last_active":"2024-04-26T20:10:13.764076+0000","last_peered":"2024-04-26T20:10:13.764076+0000","last_clean":"2024-04-26T20:10:13.764076+0000","last_became_active":"2024-04-26T20:08:16.122804+0000","last_became_peered":"2024-04-26T20:08:16.122804+0000","last_unstale":"2024-04-26T20:10:13.764076+0000","last_undegraded":"2024-04-26T20:10:13.764076+0000","last_fullsized":"2024-04-26T20:10:13.764076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:25.235296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767990+0000","last_change":"2024-04-26T20:08:17.641023+0000","last_active":"2024-04-26T20:10:13.767990+0000","last_peered":"2024-04-26T20:10:13.767990+0000","last_clean":"2024-04-26T20:10:13.767990+0000","last_became_active":"2024-04-26T20:08:17.640679+0000","last_became_peered":"2024-04-26T20:08:17.640679+0000","last_unstale":"2024-04-26T20:10:13.767990+0000","last_undegraded":"2024-04-26T20:10:13.767990+0000","last_fullsized":"2024-04-26T20:10:13.767990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:16:37.898958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350108+0000","last_change":"2024-04-26T20:08:25.704251+0000","last_active":"2024-04-26T20:10:14.350108+0000","last_peered":"2024-04-26T20:10:14.350108+0000","last_clean":"2024-04-26T20:10:14.350108+0000","last_became_active":"2024-04-26T20:08:25.702249+0000","last_became_peered":"2024-04-26T20:08:25.702249+0000","last_unstale":"2024-04-26T20:10:14.350108+0000","last_undegraded":"2024-04-26T20:10:14.350108+0000","last_fullsized":"2024-04-26T20:10:14.350108+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:03:12.107793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766554+0000","last_change":"2024-04-26T20:08:15.636474+0000","last_active":"2024-04-26T20:10:13.766554+0000","last_peered":"2024-04-26T20:10:13.766554+0000","last_clean":"2024-04-26T20:10:13.766554+0000","last_became_active":"2024-04-26T20:08:15.636104+0000","last_became_peered":"2024-04-26T20:08:15.636104+0000","last_unstale":"2024-04-26T20:10:13.766554+0000","last_undegraded":"2024-04-26T20:10:13.766554+0000","last_fullsized":"2024-04-26T20:10:13.766554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:09.514033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767700+0000","last_change":"2024-04-26T20:08:17.665683+0000","last_active":"2024-04-26T20:10:13.767700+0000","last_peered":"2024-04-26T20:10:13.767700+0000","last_clean":"2024-04-26T20:10:13.767700+0000","last_became_active":"2024-04-26T20:08:17.665385+0000","last_became_peered":"2024-04-26T20:08:17.665385+0000","last_unstale":"2024-04-26T20:10:13.767700+0000","last_undegraded":"2024-04-26T20:10:13.767700+0000","last_fullsized":"2024-04-26T20:10:13.767700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:00.826582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765673+0000","last_change":"2024-04-26T20:08:25.683715+0000","last_active":"2024-04-26T20:10:13.765673+0000","last_peered":"2024-04-26T20:10:13.765673+0000","last_clean":"2024-04-26T20:10:13.765673+0000","last_became_active":"2024-04-26T20:08:25.683155+0000","last_became_peered":"2024-04-26T20:08:25.683155+0000","last_unstale":"2024-04-26T20:10:13.765673+0000","last_undegraded":"2024-04-26T20:10:13.765673+0000","last_fullsized":"2024-04-26T20:10:13.765673+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:44.751519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":51,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.356063+0000","last_change":"2024-04-26T20:10:14.355138+0000","last_active":"2024-04-26T20:10:14.356063+0000","last_peered":"2024-04-26T20:10:14.356063+0000","last_clean":"2024-04-26T20:10:14.356063+0000","last_became_active":"2024-04-26T20:10:14.349682+0000","last_became_peered":"2024-04-26T20:10:14.349682+0000","last_unstale":"2024-04-26T20:10:14.356063+0000","last_undegraded":"2024-04-26T20:10:14.356063+0000","last_fullsized":"2024-04-26T20:10:14.356063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:46:56.365062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767978+0000","last_change":"2024-04-26T20:08:17.642480+0000","last_active":"2024-04-26T20:10:13.767978+0000","last_peered":"2024-04-26T20:10:13.767978+0000","last_clean":"2024-04-26T20:10:13.767978+0000","last_became_active":"2024-04-26T20:08:17.642165+0000","last_became_peered":"2024-04-26T20:08:17.642165+0000","last_unstale":"2024-04-26T20:10:13.767978+0000","last_undegraded":"2024-04-26T20:10:13.767978+0000","last_fullsized":"2024-04-26T20:10:13.767978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:05:39.440740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765537+0000","last_change":"2024-04-26T20:08:17.645823+0000","last_active":"2024-04-26T20:10:13.765537+0000","last_peered":"2024-04-26T20:10:13.765537+0000","last_clean":"2024-04-26T20:10:13.765537+0000","last_became_active":"2024-04-26T20:08:17.645413+0000","last_became_peered":"2024-04-26T20:08:17.645413+0000","last_unstale":"2024-04-26T20:10:13.765537+0000","last_undegraded":"2024-04-26T20:10:13.765537+0000","last_fullsized":"2024-04-26T20:10:13.765537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:36:36.622579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763916+0000","last_change":"2024-04-26T20:08:17.641413+0000","last_active":"2024-04-26T20:10:13.763916+0000","last_peered":"2024-04-26T20:10:13.763916+0000","last_clean":"2024-04-26T20:10:13.763916+0000","last_became_active":"2024-04-26T20:08:17.641073+0000","last_became_peered":"2024-04-26T20:08:17.641073+0000","last_unstale":"2024-04-26T20:10:13.763916+0000","last_undegraded":"2024-04-26T20:10:13.763916+0000","last_fullsized":"2024-04-26T20:10:13.763916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:25:03.961399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763724+0000","last_change":"2024-04-26T20:08:17.657336+0000","last_active":"2024-04-26T20:10:13.763724+0000","last_peered":"2024-04-26T20:10:13.763724+0000","last_clean":"2024-04-26T20:10:13.763724+0000","last_became_active":"2024-04-26T20:08:17.656982+0000","last_became_peered":"2024-04-26T20:08:17.656982+0000","last_unstale":"2024-04-26T20:10:13.763724+0000","last_undegraded":"2024-04-26T20:10:13.763724+0000","last_fullsized":"2024-04-26T20:10:13.763724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:10.149705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350357+0000","last_change":"2024-04-26T20:08:17.662290+0000","last_active":"2024-04-26T20:10:14.350357+0000","last_peered":"2024-04-26T20:10:14.350357+0000","last_clean":"2024-04-26T20:10:14.350357+0000","last_became_active":"2024-04-26T20:08:17.660429+0000","last_became_peered":"2024-04-26T20:08:17.660429+0000","last_unstale":"2024-04-26T20:10:14.350357+0000","last_undegraded":"2024-04-26T20:10:14.350357+0000","last_fullsized":"2024-04-26T20:10:14.350357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:50:03.563066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768324+0000","last_change":"2024-04-26T20:09:13.284038+0000","last_active":"2024-04-26T20:10:13.768324+0000","last_peered":"2024-04-26T20:10:13.768324+0000","last_clean":"2024-04-26T20:10:13.768324+0000","last_became_active":"2024-04-26T20:09:13.283650+0000","last_became_peered":"2024-04-26T20:09:13.283650+0000","last_unstale":"2024-04-26T20:10:13.768324+0000","last_undegraded":"2024-04-26T20:10:13.768324+0000","last_fullsized":"2024-04-26T20:10:13.768324+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:33.570782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766369+0000","last_change":"2024-04-26T20:08:17.642813+0000","last_active":"2024-04-26T20:10:13.766369+0000","last_peered":"2024-04-26T20:10:13.766369+0000","last_clean":"2024-04-26T20:10:13.766369+0000","last_became_active":"2024-04-26T20:08:17.642018+0000","last_became_peered":"2024-04-26T20:08:17.642018+0000","last_unstale":"2024-04-26T20:10:13.766369+0000","last_undegraded":"2024-04-26T20:10:13.766369+0000","last_fullsized":"2024-04-26T20:10:13.766369+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:19.029810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764452+0000","last_change":"2024-04-26T20:08:17.639030+0000","last_active":"2024-04-26T20:10:13.764452+0000","last_peered":"2024-04-26T20:10:13.764452+0000","last_clean":"2024-04-26T20:10:13.764452+0000","last_became_active":"2024-04-26T20:08:17.638645+0000","last_became_peered":"2024-04-26T20:08:17.638645+0000","last_unstale":"2024-04-26T20:10:13.764452+0000","last_undegraded":"2024-04-26T20:10:13.764452+0000","last_fullsized":"2024-04-26T20:10:13.764452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:58.373648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765398+0000","last_change":"2024-04-26T20:08:17.666873+0000","last_active":"2024-04-26T20:10:13.765398+0000","last_peered":"2024-04-26T20:10:13.765398+0000","last_clean":"2024-04-26T20:10:13.765398+0000","last_became_active":"2024-04-26T20:08:17.666185+0000","last_became_peered":"2024-04-26T20:08:17.666185+0000","last_unstale":"2024-04-26T20:10:13.765398+0000","last_undegraded":"2024-04-26T20:10:13.765398+0000","last_fullsized":"2024-04-26T20:10:13.765398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:51:17.105637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765928+0000","last_change":"2024-04-26T20:08:17.666625+0000","last_active":"2024-04-26T20:10:13.765928+0000","last_peered":"2024-04-26T20:10:13.765928+0000","last_clean":"2024-04-26T20:10:13.765928+0000","last_became_active":"2024-04-26T20:08:17.666288+0000","last_became_peered":"2024-04-26T20:08:17.666288+0000","last_unstale":"2024-04-26T20:10:13.765928+0000","last_undegraded":"2024-04-26T20:10:13.765928+0000","last_fullsized":"2024-04-26T20:10:13.765928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:27.051752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764134+0000","last_change":"2024-04-26T20:08:17.663490+0000","last_active":"2024-04-26T20:10:13.764134+0000","last_peered":"2024-04-26T20:10:13.764134+0000","last_clean":"2024-04-26T20:10:13.764134+0000","last_became_active":"2024-04-26T20:08:17.663131+0000","last_became_peered":"2024-04-26T20:08:17.663131+0000","last_unstale":"2024-04-26T20:10:13.764134+0000","last_undegraded":"2024-04-26T20:10:13.764134+0000","last_fullsized":"2024-04-26T20:10:13.764134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:46.633558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930123+0000","last_change":"2024-04-26T20:08:17.664749+0000","last_active":"2024-04-26T20:10:13.930123+0000","last_peered":"2024-04-26T20:10:13.930123+0000","last_clean":"2024-04-26T20:10:13.930123+0000","last_became_active":"2024-04-26T20:08:17.664346+0000","last_became_peered":"2024-04-26T20:08:17.664346+0000","last_unstale":"2024-04-26T20:10:13.930123+0000","last_undegraded":"2024-04-26T20:10:13.930123+0000","last_fullsized":"2024-04-26T20:10:13.930123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:27.106743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767911+0000","last_change":"2024-04-26T20:08:17.662955+0000","last_active":"2024-04-26T20:10:13.767911+0000","last_peered":"2024-04-26T20:10:13.767911+0000","last_clean":"2024-04-26T20:10:13.767911+0000","last_became_active":"2024-04-26T20:08:17.662639+0000","last_became_peered":"2024-04-26T20:08:17.662639+0000","last_unstale":"2024-04-26T20:10:13.767911+0000","last_undegraded":"2024-04-26T20:10:13.767911+0000","last_fullsized":"2024-04-26T20:10:13.767911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:28.346696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765592+0000","last_change":"2024-04-26T20:08:17.641465+0000","last_active":"2024-04-26T20:10:13.765592+0000","last_peered":"2024-04-26T20:10:13.765592+0000","last_clean":"2024-04-26T20:10:13.765592+0000","last_became_active":"2024-04-26T20:08:17.641145+0000","last_became_peered":"2024-04-26T20:08:17.641145+0000","last_unstale":"2024-04-26T20:10:13.765592+0000","last_undegraded":"2024-04-26T20:10:13.765592+0000","last_fullsized":"2024-04-26T20:10:13.765592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:58.977174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61112,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":621,"num_read_kb":380,"num_write":169,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1595,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383731,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1101824,"data_stored":545440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645366,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28768,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716384,"statfs":{"total":95995035648,"available":95965577216,"internally_reserved":0,"allocated":1785856,"data_stored":1193333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808893,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1114112,"data_stored":545094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070531,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":545014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332169,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708144,"statfs":{"total":95995035648,"available":95957139456,"internally_reserved":0,"allocated":1703936,"data_stored":1140341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495694,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":540270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36480,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708672,"statfs":{"total":95995035648,"available":95957680128,"internally_reserved":0,"allocated":1163264,"data_stored":598304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1118208,"data_stored":541658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1171456,"data_stored":598019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182495,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":540464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920857,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":544920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757332,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28700,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716452,"statfs":{"total":95995035648,"available":95965646848,"internally_reserved":0,"allocated":1716224,"data_stored":1136511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":58124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":57839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T20:11:09.245 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T20:11:09.688 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:11:10.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:09 smithi103 ceph-mon[32635]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:11:10.076 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:09 smithi094 ceph-mon[28050]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:11:10.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:09 smithi204 ceph-mon[32618]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:11:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:10 smithi103 ceph-mon[32635]: pgmap v415: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:11.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:10 smithi204 ceph-mon[32618]: pgmap v415: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:11.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:10 smithi094 ceph-mon[28050]: pgmap v415: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:11.951 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:11 smithi094 ceph-mon[28050]: pgmap v416: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:12.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:11 smithi103 ceph-mon[32635]: pgmap v416: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:12.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:11 smithi204 ceph-mon[32618]: pgmap v416: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:12.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.213+0000 7f8ea3e00700 1 -- 172.21.15.94:0/3252622331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea40760e0 msgr2=0x7f8ea4113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:12.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.213+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/3252622331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea40760e0 0x7f8ea4113e10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c0098f0 tx=0x7f8e8c009c00 comp rx=0 tx=0).stop 2024-04-26T20:11:12.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.214+0000 7f8ea3e00700 1 -- 172.21.15.94:0/3252622331 shutdown_connections 2024-04-26T20:11:12.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.214+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/3252622331 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ea4114350 0x7f8ea4116740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:12.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.214+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/3252622331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ea40757c0 0x7f8ea4075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.214+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/3252622331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea40760e0 0x7f8ea4113e10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:12.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.214+0000 7f8ea3e00700 1 -- 172.21.15.94:0/3252622331 >> 172.21.15.94:0/3252622331 conn(0x7f8ea40fee30 msgr2=0x7f8ea4101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:12.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.214+0000 7f8ea3e00700 1 -- 172.21.15.94:0/3252622331 shutdown_connections 2024-04-26T20:11:12.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.214+0000 7f8ea3e00700 1 -- 172.21.15.94:0/3252622331 wait complete. 2024-04-26T20:11:12.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.215+0000 7f8ea3e00700 1 Processor -- start 2024-04-26T20:11:12.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.215+0000 7f8ea3e00700 1 -- start start 2024-04-26T20:11:12.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.215+0000 7f8ea3e00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ea40757c0 0x7f8ea41a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:12.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.215+0000 7f8ea3e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ea40760e0 0x7f8ea41a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:12.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.215+0000 7f8ea3e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea4114350 0x7f8ea41a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.215+0000 7f8ea3e00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea4119b70 con 0x7f8ea40760e0 2024-04-26T20:11:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.215+0000 7f8ea3e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41199f0 con 0x7f8ea4114350 2024-04-26T20:11:12.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.215+0000 7f8ea3e00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea4119cf0 con 0x7f8ea40757c0 2024-04-26T20:11:12.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8ea2a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ea40757c0 0x7f8ea41a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:12.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8e9a000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ea40760e0 0x7f8ea41a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:12.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8ea2a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ea40757c0 0x7f8ea41a00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:55110/0 (socket says 172.21.15.94:55110) 2024-04-26T20:11:12.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8ea2a00700 1 -- 172.21.15.94:0/2941757020 learned_addr learned my addr 172.21.15.94:0/2941757020 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:12.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8ea3400700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea4114350 0x7f8ea41a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:12.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8ea2a00700 1 -- 172.21.15.94:0/2941757020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea4114350 msgr2=0x7f8ea41a49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:12.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8ea2a00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea4114350 0x7f8ea41a49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:12.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8ea2a00700 1 -- 172.21.15.94:0/2941757020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ea40760e0 msgr2=0x7f8ea41a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8ea2a00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ea40760e0 0x7f8ea41a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:12.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.216+0000 7f8ea2a00700 1 -- 172.21.15.94:0/2941757020 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea41a50f0 con 0x7f8ea40757c0 2024-04-26T20:11:12.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.217+0000 7f8ea2a00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ea40757c0 0x7f8ea41a00e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400ef10 tx=0x7f8e9400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:12.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.217+0000 7f8e9be00700 1 -- 172.21.15.94:0/2941757020 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9400c9b0 con 0x7f8ea40757c0 2024-04-26T20:11:12.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.217+0000 7f8e9be00700 1 -- 172.21.15.94:0/2941757020 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e94019070 con 0x7f8ea40757c0 2024-04-26T20:11:12.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.217+0000 7f8e9be00700 1 -- 172.21.15.94:0/2941757020 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e94014480 con 0x7f8ea40757c0 2024-04-26T20:11:12.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.217+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea41a5380 con 0x7f8ea40757c0 2024-04-26T20:11:12.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.217+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea41acc60 con 0x7f8ea40757c0 2024-04-26T20:11:12.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.218+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea406a110 con 0x7f8ea40757c0 2024-04-26T20:11:12.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.220+0000 7f8e9be00700 1 -- 172.21.15.94:0/2941757020 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8e94012050 con 0x7f8ea40757c0 2024-04-26T20:11:12.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.221+0000 7f8e9be00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e880a3d50 0x7f8e880a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:12.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.222+0000 7f8e9be00700 1 -- 172.21.15.94:0/2941757020 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f8e940d0ef0 con 0x7f8ea40757c0 2024-04-26T20:11:12.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.222+0000 7f8e9a000700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e880a3d50 0x7f8e880a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:12.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.223+0000 7f8e9a000700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e880a3d50 0x7f8e880a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c0098c0 tx=0x7f8e8c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:12.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.224+0000 7f8e9be00700 1 -- 172.21.15.94:0/2941757020 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8e94010040 con 0x7f8ea40757c0 2024-04-26T20:11:13.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.443+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8ea4063860 con 0x7f8e880a3d50 2024-04-26T20:11:13.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.460+0000 7f8e9be00700 1 -- 172.21.15.94:0/2941757020 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340382 (secure 0 0 0) 0x7f8e8c07a1b0 con 0x7f8e880a3d50 2024-04-26T20:11:13.075 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:12 smithi103 ceph-mon[32635]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:11:13.087 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:11:13.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e880a3d50 msgr2=0x7f8e880a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:13.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e880a3d50 0x7f8e880a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c0098c0 tx=0x7f8e8c019040 comp rx=0 tx=0).stop 2024-04-26T20:11:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ea40757c0 msgr2=0x7f8ea41a00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:13.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ea40757c0 0x7f8ea41a00e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400ef10 tx=0x7f8e9400c3a0 comp rx=0 tx=0).stop 2024-04-26T20:11:13.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 shutdown_connections 2024-04-26T20:11:13.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e880a3d50 0x7f8e880a6210 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:13.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea4114350 0x7f8ea41a49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:13.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ea40757c0 0x7f8ea41a00e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:13.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 --2- 172.21.15.94:0/2941757020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ea40760e0 0x7f8ea41a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 >> 172.21.15.94:0/2941757020 conn(0x7f8ea40fee30 msgr2=0x7f8ea4100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:13.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 shutdown_connections 2024-04-26T20:11:13.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:12.468+0000 7f8ea3e00700 1 -- 172.21.15.94:0/2941757020 wait complete. 2024-04-26T20:11:13.235 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-04-26T20:11:13.294 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:12 smithi094 ceph-mon[28050]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:11:13.300 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:12 smithi204 ceph-mon[32618]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:11:13.349 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-04-26T20:11:11.631209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651714,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":772,"num_read_kb":499,"num_write":352,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1182331,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":397448,"kb_used_data":14936,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124544376,"statfs":{"total":1151940427776,"available":1151533441024,"internally_reserved":0,"allocated":15294464,"data_stored":8469368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":391492924},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004500"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766125+0000","last_change":"2024-04-26T20:08:17.653922+0000","last_active":"2024-04-26T20:10:13.766125+0000","last_peered":"2024-04-26T20:10:13.766125+0000","last_clean":"2024-04-26T20:10:13.766125+0000","last_became_active":"2024-04-26T20:08:17.653486+0000","last_became_peered":"2024-04-26T20:08:17.653486+0000","last_unstale":"2024-04-26T20:10:13.766125+0000","last_undegraded":"2024-04-26T20:10:13.766125+0000","last_fullsized":"2024-04-26T20:10:13.766125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:05.034829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350663+0000","last_change":"2024-04-26T20:08:17.650126+0000","last_active":"2024-04-26T20:10:14.350663+0000","last_peered":"2024-04-26T20:10:14.350663+0000","last_clean":"2024-04-26T20:10:14.350663+0000","last_became_active":"2024-04-26T20:08:17.649727+0000","last_became_peered":"2024-04-26T20:08:17.649727+0000","last_unstale":"2024-04-26T20:10:14.350663+0000","last_undegraded":"2024-04-26T20:10:14.350663+0000","last_fullsized":"2024-04-26T20:10:14.350663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:11:28.255286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350429+0000","last_change":"2024-04-26T20:08:17.646243+0000","last_active":"2024-04-26T20:10:14.350429+0000","last_peered":"2024-04-26T20:10:14.350429+0000","last_clean":"2024-04-26T20:10:14.350429+0000","last_became_active":"2024-04-26T20:08:17.645677+0000","last_became_peered":"2024-04-26T20:08:17.645677+0000","last_unstale":"2024-04-26T20:10:14.350429+0000","last_undegraded":"2024-04-26T20:10:14.350429+0000","last_fullsized":"2024-04-26T20:10:14.350429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:31.698329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347798+0000","last_change":"2024-04-26T20:08:17.639845+0000","last_active":"2024-04-26T20:10:14.347798+0000","last_peered":"2024-04-26T20:10:14.347798+0000","last_clean":"2024-04-26T20:10:14.347798+0000","last_became_active":"2024-04-26T20:08:17.639465+0000","last_became_peered":"2024-04-26T20:08:17.639465+0000","last_unstale":"2024-04-26T20:10:14.347798+0000","last_undegraded":"2024-04-26T20:10:14.347798+0000","last_fullsized":"2024-04-26T20:10:14.347798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:05.717196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768688+0000","last_change":"2024-04-26T20:08:17.660459+0000","last_active":"2024-04-26T20:10:13.768688+0000","last_peered":"2024-04-26T20:10:13.768688+0000","last_clean":"2024-04-26T20:10:13.768688+0000","last_became_active":"2024-04-26T20:08:17.659927+0000","last_became_peered":"2024-04-26T20:08:17.659927+0000","last_unstale":"2024-04-26T20:10:13.768688+0000","last_undegraded":"2024-04-26T20:10:13.768688+0000","last_fullsized":"2024-04-26T20:10:13.768688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:10.062073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763490+0000","last_change":"2024-04-26T20:09:13.279825+0000","last_active":"2024-04-26T20:10:13.763490+0000","last_peered":"2024-04-26T20:10:13.763490+0000","last_clean":"2024-04-26T20:10:13.763490+0000","last_became_active":"2024-04-26T20:09:13.279464+0000","last_became_peered":"2024-04-26T20:09:13.279464+0000","last_unstale":"2024-04-26T20:10:13.763490+0000","last_undegraded":"2024-04-26T20:10:13.763490+0000","last_fullsized":"2024-04-26T20:10:13.763490+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:41.687987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768026+0000","last_change":"2024-04-26T20:08:17.648806+0000","last_active":"2024-04-26T20:10:13.768026+0000","last_peered":"2024-04-26T20:10:13.768026+0000","last_clean":"2024-04-26T20:10:13.768026+0000","last_became_active":"2024-04-26T20:08:17.648552+0000","last_became_peered":"2024-04-26T20:08:17.648552+0000","last_unstale":"2024-04-26T20:10:13.768026+0000","last_undegraded":"2024-04-26T20:10:13.768026+0000","last_fullsized":"2024-04-26T20:10:13.768026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:43.282610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763953+0000","last_change":"2024-04-26T20:08:17.663479+0000","last_active":"2024-04-26T20:10:13.763953+0000","last_peered":"2024-04-26T20:10:13.763953+0000","last_clean":"2024-04-26T20:10:13.763953+0000","last_became_active":"2024-04-26T20:08:17.663043+0000","last_became_peered":"2024-04-26T20:08:17.663043+0000","last_unstale":"2024-04-26T20:10:13.763953+0000","last_undegraded":"2024-04-26T20:10:13.763953+0000","last_fullsized":"2024-04-26T20:10:13.763953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:21:36.518513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930272+0000","last_change":"2024-04-26T20:08:17.637978+0000","last_active":"2024-04-26T20:10:13.930272+0000","last_peered":"2024-04-26T20:10:13.930272+0000","last_clean":"2024-04-26T20:10:13.930272+0000","last_became_active":"2024-04-26T20:08:17.637458+0000","last_became_peered":"2024-04-26T20:08:17.637458+0000","last_unstale":"2024-04-26T20:10:13.930272+0000","last_undegraded":"2024-04-26T20:10:13.930272+0000","last_fullsized":"2024-04-26T20:10:13.930272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:20:01.232741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763731+0000","last_change":"2024-04-26T20:08:17.639819+0000","last_active":"2024-04-26T20:10:13.763731+0000","last_peered":"2024-04-26T20:10:13.763731+0000","last_clean":"2024-04-26T20:10:13.763731+0000","last_became_active":"2024-04-26T20:08:17.639479+0000","last_became_peered":"2024-04-26T20:08:17.639479+0000","last_unstale":"2024-04-26T20:10:13.763731+0000","last_undegraded":"2024-04-26T20:10:13.763731+0000","last_fullsized":"2024-04-26T20:10:13.763731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:44.536585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767910+0000","last_change":"2024-04-26T20:08:17.642533+0000","last_active":"2024-04-26T20:10:13.767910+0000","last_peered":"2024-04-26T20:10:13.767910+0000","last_clean":"2024-04-26T20:10:13.767910+0000","last_became_active":"2024-04-26T20:08:17.642198+0000","last_became_peered":"2024-04-26T20:08:17.642198+0000","last_unstale":"2024-04-26T20:10:13.767910+0000","last_undegraded":"2024-04-26T20:10:13.767910+0000","last_fullsized":"2024-04-26T20:10:13.767910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:16:21.200183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347765+0000","last_change":"2024-04-26T20:08:17.644646+0000","last_active":"2024-04-26T20:10:14.347765+0000","last_peered":"2024-04-26T20:10:14.347765+0000","last_clean":"2024-04-26T20:10:14.347765+0000","last_became_active":"2024-04-26T20:08:17.644314+0000","last_became_peered":"2024-04-26T20:08:17.644314+0000","last_unstale":"2024-04-26T20:10:14.347765+0000","last_undegraded":"2024-04-26T20:10:14.347765+0000","last_fullsized":"2024-04-26T20:10:14.347765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:55:00.003120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767886+0000","last_change":"2024-04-26T20:08:17.657118+0000","last_active":"2024-04-26T20:10:13.767886+0000","last_peered":"2024-04-26T20:10:13.767886+0000","last_clean":"2024-04-26T20:10:13.767886+0000","last_became_active":"2024-04-26T20:08:17.656256+0000","last_became_peered":"2024-04-26T20:08:17.656256+0000","last_unstale":"2024-04-26T20:10:13.767886+0000","last_undegraded":"2024-04-26T20:10:13.767886+0000","last_fullsized":"2024-04-26T20:10:13.767886+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:09.133475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.348535+0000","last_change":"2024-04-26T20:08:17.658234+0000","last_active":"2024-04-26T20:10:14.348535+0000","last_peered":"2024-04-26T20:10:14.348535+0000","last_clean":"2024-04-26T20:10:14.348535+0000","last_became_active":"2024-04-26T20:08:17.651714+0000","last_became_peered":"2024-04-26T20:08:17.651714+0000","last_unstale":"2024-04-26T20:10:14.348535+0000","last_undegraded":"2024-04-26T20:10:14.348535+0000","last_fullsized":"2024-04-26T20:10:14.348535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:30.630320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.348612+0000","last_change":"2024-04-26T20:08:17.647898+0000","last_active":"2024-04-26T20:10:14.348612+0000","last_peered":"2024-04-26T20:10:14.348612+0000","last_clean":"2024-04-26T20:10:14.348612+0000","last_became_active":"2024-04-26T20:08:17.647463+0000","last_became_peered":"2024-04-26T20:08:17.647463+0000","last_unstale":"2024-04-26T20:10:14.348612+0000","last_undegraded":"2024-04-26T20:10:14.348612+0000","last_fullsized":"2024-04-26T20:10:14.348612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:24.180974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.929767+0000","last_change":"2024-04-26T20:08:17.639518+0000","last_active":"2024-04-26T20:10:13.929767+0000","last_peered":"2024-04-26T20:10:13.929767+0000","last_clean":"2024-04-26T20:10:13.929767+0000","last_became_active":"2024-04-26T20:08:17.639183+0000","last_became_peered":"2024-04-26T20:08:17.639183+0000","last_unstale":"2024-04-26T20:10:13.929767+0000","last_undegraded":"2024-04-26T20:10:13.929767+0000","last_fullsized":"2024-04-26T20:10:13.929767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:41:12.893323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765922+0000","last_change":"2024-04-26T20:08:17.653956+0000","last_active":"2024-04-26T20:10:13.765922+0000","last_peered":"2024-04-26T20:10:13.765922+0000","last_clean":"2024-04-26T20:10:13.765922+0000","last_became_active":"2024-04-26T20:08:17.653576+0000","last_became_peered":"2024-04-26T20:08:17.653576+0000","last_unstale":"2024-04-26T20:10:13.765922+0000","last_undegraded":"2024-04-26T20:10:13.765922+0000","last_fullsized":"2024-04-26T20:10:13.765922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:35:09.538194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770895+0000","last_change":"2024-04-26T20:08:17.649500+0000","last_active":"2024-04-26T20:10:13.770895+0000","last_peered":"2024-04-26T20:10:13.770895+0000","last_clean":"2024-04-26T20:10:13.770895+0000","last_became_active":"2024-04-26T20:08:17.649062+0000","last_became_peered":"2024-04-26T20:08:17.649062+0000","last_unstale":"2024-04-26T20:10:13.770895+0000","last_undegraded":"2024-04-26T20:10:13.770895+0000","last_fullsized":"2024-04-26T20:10:13.770895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:24.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764212+0000","last_change":"2024-04-26T20:08:17.666440+0000","last_active":"2024-04-26T20:10:13.764212+0000","last_peered":"2024-04-26T20:10:13.764212+0000","last_clean":"2024-04-26T20:10:13.764212+0000","last_became_active":"2024-04-26T20:08:17.666024+0000","last_became_peered":"2024-04-26T20:08:17.666024+0000","last_unstale":"2024-04-26T20:10:13.764212+0000","last_undegraded":"2024-04-26T20:10:13.764212+0000","last_fullsized":"2024-04-26T20:10:13.764212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:55.356424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763679+0000","last_change":"2024-04-26T20:08:25.700728+0000","last_active":"2024-04-26T20:10:13.763679+0000","last_peered":"2024-04-26T20:10:13.763679+0000","last_clean":"2024-04-26T20:10:13.763679+0000","last_became_active":"2024-04-26T20:08:25.700339+0000","last_became_peered":"2024-04-26T20:08:25.700339+0000","last_unstale":"2024-04-26T20:10:13.763679+0000","last_undegraded":"2024-04-26T20:10:13.763679+0000","last_fullsized":"2024-04-26T20:10:13.763679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:53.581083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'9","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.778543+0000","last_change":"2024-04-26T20:10:13.778038+0000","last_active":"2024-04-26T20:10:13.778543+0000","last_peered":"2024-04-26T20:10:13.778543+0000","last_clean":"2024-04-26T20:10:13.778543+0000","last_became_active":"2024-04-26T20:10:13.772007+0000","last_became_peered":"2024-04-26T20:10:13.772007+0000","last_unstale":"2024-04-26T20:10:13.778543+0000","last_undegraded":"2024-04-26T20:10:13.778543+0000","last_fullsized":"2024-04-26T20:10:13.778543+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:05:26.873150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764986+0000","last_change":"2024-04-26T20:08:17.657637+0000","last_active":"2024-04-26T20:10:13.764986+0000","last_peered":"2024-04-26T20:10:13.764986+0000","last_clean":"2024-04-26T20:10:13.764986+0000","last_became_active":"2024-04-26T20:08:17.657290+0000","last_became_peered":"2024-04-26T20:08:17.657290+0000","last_unstale":"2024-04-26T20:10:13.764986+0000","last_undegraded":"2024-04-26T20:10:13.764986+0000","last_fullsized":"2024-04-26T20:10:13.764986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:30:25.270989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.346942+0000","last_change":"2024-04-26T20:09:13.318084+0000","last_active":"2024-04-26T20:10:14.346942+0000","last_peered":"2024-04-26T20:10:14.346942+0000","last_clean":"2024-04-26T20:10:14.346942+0000","last_became_active":"2024-04-26T20:09:13.317689+0000","last_became_peered":"2024-04-26T20:09:13.317689+0000","last_unstale":"2024-04-26T20:10:14.346942+0000","last_undegraded":"2024-04-26T20:10:14.346942+0000","last_fullsized":"2024-04-26T20:10:14.346942+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:25:48.182186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"94'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765720+0000","last_change":"2024-04-26T20:08:16.128717+0000","last_active":"2024-04-26T20:10:13.765720+0000","last_peered":"2024-04-26T20:10:13.765720+0000","last_clean":"2024-04-26T20:10:13.765720+0000","last_became_active":"2024-04-26T20:08:16.127738+0000","last_became_peered":"2024-04-26T20:08:16.127738+0000","last_unstale":"2024-04-26T20:10:13.765720+0000","last_undegraded":"2024-04-26T20:10:13.765720+0000","last_fullsized":"2024-04-26T20:10:13.765720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:01:26.343914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350943+0000","last_change":"2024-04-26T20:08:17.659613+0000","last_active":"2024-04-26T20:10:14.350943+0000","last_peered":"2024-04-26T20:10:14.350943+0000","last_clean":"2024-04-26T20:10:14.350943+0000","last_became_active":"2024-04-26T20:08:17.658750+0000","last_became_peered":"2024-04-26T20:08:17.658750+0000","last_unstale":"2024-04-26T20:10:14.350943+0000","last_undegraded":"2024-04-26T20:10:14.350943+0000","last_fullsized":"2024-04-26T20:10:14.350943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:59:08.724656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347077+0000","last_change":"2024-04-26T20:08:25.709115+0000","last_active":"2024-04-26T20:10:14.347077+0000","last_peered":"2024-04-26T20:10:14.347077+0000","last_clean":"2024-04-26T20:10:14.347077+0000","last_became_active":"2024-04-26T20:08:25.708552+0000","last_became_peered":"2024-04-26T20:08:25.708552+0000","last_unstale":"2024-04-26T20:10:14.347077+0000","last_undegraded":"2024-04-26T20:10:14.347077+0000","last_fullsized":"2024-04-26T20:10:14.347077+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:20.735727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"94'4","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768027+0000","last_change":"2024-04-26T20:08:15.624441+0000","last_active":"2024-04-26T20:10:13.768027+0000","last_peered":"2024-04-26T20:10:13.768027+0000","last_clean":"2024-04-26T20:10:13.768027+0000","last_became_active":"2024-04-26T20:08:15.623999+0000","last_became_peered":"2024-04-26T20:08:15.623999+0000","last_unstale":"2024-04-26T20:10:13.768027+0000","last_undegraded":"2024-04-26T20:10:13.768027+0000","last_fullsized":"2024-04-26T20:10:13.768027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:34.276360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930010+0000","last_change":"2024-04-26T20:08:17.656088+0000","last_active":"2024-04-26T20:10:13.930010+0000","last_peered":"2024-04-26T20:10:13.930010+0000","last_clean":"2024-04-26T20:10:13.930010+0000","last_became_active":"2024-04-26T20:08:17.655736+0000","last_became_peered":"2024-04-26T20:08:17.655736+0000","last_unstale":"2024-04-26T20:10:13.930010+0000","last_undegraded":"2024-04-26T20:10:13.930010+0000","last_fullsized":"2024-04-26T20:10:13.930010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:29.187893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.346963+0000","last_change":"2024-04-26T20:08:25.703015+0000","last_active":"2024-04-26T20:10:14.346963+0000","last_peered":"2024-04-26T20:10:14.346963+0000","last_clean":"2024-04-26T20:10:14.346963+0000","last_became_active":"2024-04-26T20:08:25.702699+0000","last_became_peered":"2024-04-26T20:08:25.702699+0000","last_unstale":"2024-04-26T20:10:14.346963+0000","last_undegraded":"2024-04-26T20:10:14.346963+0000","last_fullsized":"2024-04-26T20:10:14.346963+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:29:17.082803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.769010+0000","last_change":"2024-04-26T20:08:15.624092+0000","last_active":"2024-04-26T20:10:13.769010+0000","last_peered":"2024-04-26T20:10:13.769010+0000","last_clean":"2024-04-26T20:10:13.769010+0000","last_became_active":"2024-04-26T20:08:15.623540+0000","last_became_peered":"2024-04-26T20:08:15.623540+0000","last_unstale":"2024-04-26T20:10:13.769010+0000","last_undegraded":"2024-04-26T20:10:13.769010+0000","last_fullsized":"2024-04-26T20:10:13.769010+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:06:17.162854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766456+0000","last_change":"2024-04-26T20:09:13.313449+0000","last_active":"2024-04-26T20:10:13.766456+0000","last_peered":"2024-04-26T20:10:13.766456+0000","last_clean":"2024-04-26T20:10:13.766456+0000","last_became_active":"2024-04-26T20:09:13.313124+0000","last_became_peered":"2024-04-26T20:09:13.313124+0000","last_unstale":"2024-04-26T20:10:13.766456+0000","last_undegraded":"2024-04-26T20:10:13.766456+0000","last_fullsized":"2024-04-26T20:10:13.766456+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:10.447987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763999+0000","last_change":"2024-04-26T20:08:25.684973+0000","last_active":"2024-04-26T20:10:13.763999+0000","last_peered":"2024-04-26T20:10:13.763999+0000","last_clean":"2024-04-26T20:10:13.763999+0000","last_became_active":"2024-04-26T20:08:25.684636+0000","last_became_peered":"2024-04-26T20:08:25.684636+0000","last_unstale":"2024-04-26T20:10:13.763999+0000","last_undegraded":"2024-04-26T20:10:13.763999+0000","last_fullsized":"2024-04-26T20:10:13.763999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:47:17.140765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":163,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:06.004084+0000","last_change":"2024-04-26T20:10:13.934713+0000","last_active":"2024-04-26T20:11:06.004084+0000","last_peered":"2024-04-26T20:11:06.004084+0000","last_clean":"2024-04-26T20:11:06.004084+0000","last_became_active":"2024-04-26T20:10:13.930796+0000","last_became_peered":"2024-04-26T20:10:13.930796+0000","last_unstale":"2024-04-26T20:11:06.004084+0000","last_undegraded":"2024-04-26T20:11:06.004084+0000","last_fullsized":"2024-04-26T20:11:06.004084+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:42:28.016674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768580+0000","last_change":"2024-04-26T20:08:17.660345+0000","last_active":"2024-04-26T20:10:13.768580+0000","last_peered":"2024-04-26T20:10:13.768580+0000","last_clean":"2024-04-26T20:10:13.768580+0000","last_became_active":"2024-04-26T20:08:17.659966+0000","last_became_peered":"2024-04-26T20:08:17.659966+0000","last_unstale":"2024-04-26T20:10:13.768580+0000","last_undegraded":"2024-04-26T20:10:13.768580+0000","last_fullsized":"2024-04-26T20:10:13.768580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:48.762984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763965+0000","last_change":"2024-04-26T20:08:25.698002+0000","last_active":"2024-04-26T20:10:13.763965+0000","last_peered":"2024-04-26T20:10:13.763965+0000","last_clean":"2024-04-26T20:10:13.763965+0000","last_became_active":"2024-04-26T20:08:25.697588+0000","last_became_peered":"2024-04-26T20:08:25.697588+0000","last_unstale":"2024-04-26T20:10:13.763965+0000","last_undegraded":"2024-04-26T20:10:13.763965+0000","last_fullsized":"2024-04-26T20:10:13.763965+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:59.771941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"94'3","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763425+0000","last_change":"2024-04-26T20:08:15.633759+0000","last_active":"2024-04-26T20:10:13.763425+0000","last_peered":"2024-04-26T20:10:13.763425+0000","last_clean":"2024-04-26T20:10:13.763425+0000","last_became_active":"2024-04-26T20:08:15.633354+0000","last_became_peered":"2024-04-26T20:08:15.633354+0000","last_unstale":"2024-04-26T20:10:13.763425+0000","last_undegraded":"2024-04-26T20:10:13.763425+0000","last_fullsized":"2024-04-26T20:10:13.763425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:31:33.394418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.769795+0000","last_change":"2024-04-26T20:09:13.319471+0000","last_active":"2024-04-26T20:10:13.769795+0000","last_peered":"2024-04-26T20:10:13.769795+0000","last_clean":"2024-04-26T20:10:13.769795+0000","last_became_active":"2024-04-26T20:09:13.318292+0000","last_became_peered":"2024-04-26T20:09:13.318292+0000","last_unstale":"2024-04-26T20:10:13.769795+0000","last_undegraded":"2024-04-26T20:10:13.769795+0000","last_fullsized":"2024-04-26T20:10:13.769795+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:41:36.278298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765836+0000","last_change":"2024-04-26T20:08:25.713338+0000","last_active":"2024-04-26T20:10:13.765836+0000","last_peered":"2024-04-26T20:10:13.765836+0000","last_clean":"2024-04-26T20:10:13.765836+0000","last_became_active":"2024-04-26T20:08:25.712905+0000","last_became_peered":"2024-04-26T20:08:25.712905+0000","last_unstale":"2024-04-26T20:10:13.765836+0000","last_undegraded":"2024-04-26T20:10:13.765836+0000","last_fullsized":"2024-04-26T20:10:13.765836+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:14:54.811957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"94'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767348+0000","last_change":"2024-04-26T20:08:15.633813+0000","last_active":"2024-04-26T20:10:13.767348+0000","last_peered":"2024-04-26T20:10:13.767348+0000","last_clean":"2024-04-26T20:10:13.767348+0000","last_became_active":"2024-04-26T20:08:15.633448+0000","last_became_peered":"2024-04-26T20:08:15.633448+0000","last_unstale":"2024-04-26T20:10:13.767348+0000","last_undegraded":"2024-04-26T20:10:13.767348+0000","last_fullsized":"2024-04-26T20:10:13.767348+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:52.909106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764347+0000","last_change":"2024-04-26T20:08:17.659606+0000","last_active":"2024-04-26T20:10:13.764347+0000","last_peered":"2024-04-26T20:10:13.764347+0000","last_clean":"2024-04-26T20:10:13.764347+0000","last_became_active":"2024-04-26T20:08:17.658948+0000","last_became_peered":"2024-04-26T20:08:17.658948+0000","last_unstale":"2024-04-26T20:10:13.764347+0000","last_undegraded":"2024-04-26T20:10:13.764347+0000","last_fullsized":"2024-04-26T20:10:13.764347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:03.616172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350555+0000","last_change":"2024-04-26T20:08:25.700149+0000","last_active":"2024-04-26T20:10:14.350555+0000","last_peered":"2024-04-26T20:10:14.350555+0000","last_clean":"2024-04-26T20:10:14.350555+0000","last_became_active":"2024-04-26T20:08:25.699693+0000","last_became_peered":"2024-04-26T20:08:25.699693+0000","last_unstale":"2024-04-26T20:10:14.350555+0000","last_undegraded":"2024-04-26T20:10:14.350555+0000","last_fullsized":"2024-04-26T20:10:14.350555+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:53.029443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"94'1","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765834+0000","last_change":"2024-04-26T20:08:15.626386+0000","last_active":"2024-04-26T20:10:13.765834+0000","last_peered":"2024-04-26T20:10:13.765834+0000","last_clean":"2024-04-26T20:10:13.765834+0000","last_became_active":"2024-04-26T20:08:15.626071+0000","last_became_peered":"2024-04-26T20:08:15.626071+0000","last_unstale":"2024-04-26T20:10:13.765834+0000","last_undegraded":"2024-04-26T20:10:13.765834+0000","last_fullsized":"2024-04-26T20:10:13.765834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:49.478038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770021+0000","last_change":"2024-04-26T20:08:17.668616+0000","last_active":"2024-04-26T20:10:13.770021+0000","last_peered":"2024-04-26T20:10:13.770021+0000","last_clean":"2024-04-26T20:10:13.770021+0000","last_became_active":"2024-04-26T20:08:17.668210+0000","last_became_peered":"2024-04-26T20:08:17.668210+0000","last_unstale":"2024-04-26T20:10:13.770021+0000","last_undegraded":"2024-04-26T20:10:13.770021+0000","last_fullsized":"2024-04-26T20:10:13.770021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:42.219107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763958+0000","last_change":"2024-04-26T20:08:25.699132+0000","last_active":"2024-04-26T20:10:13.763958+0000","last_peered":"2024-04-26T20:10:13.763958+0000","last_clean":"2024-04-26T20:10:13.763958+0000","last_became_active":"2024-04-26T20:08:25.698675+0000","last_became_peered":"2024-04-26T20:08:25.698675+0000","last_unstale":"2024-04-26T20:10:13.763958+0000","last_undegraded":"2024-04-26T20:10:13.763958+0000","last_fullsized":"2024-04-26T20:10:13.763958+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:08:21.950765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'3","reported_seq":147,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:08.006263+0000","last_change":"2024-04-26T20:08:15.627167+0000","last_active":"2024-04-26T20:11:08.006263+0000","last_peered":"2024-04-26T20:11:08.006263+0000","last_clean":"2024-04-26T20:11:08.006263+0000","last_became_active":"2024-04-26T20:08:15.625928+0000","last_became_peered":"2024-04-26T20:08:15.625928+0000","last_unstale":"2024-04-26T20:11:08.006263+0000","last_undegraded":"2024-04-26T20:11:08.006263+0000","last_fullsized":"2024-04-26T20:11:08.006263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:17:50.361864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930305+0000","last_change":"2024-04-26T20:08:17.652724+0000","last_active":"2024-04-26T20:10:13.930305+0000","last_peered":"2024-04-26T20:10:13.930305+0000","last_clean":"2024-04-26T20:10:13.930305+0000","last_became_active":"2024-04-26T20:08:17.652342+0000","last_became_peered":"2024-04-26T20:08:17.652342+0000","last_unstale":"2024-04-26T20:10:13.930305+0000","last_undegraded":"2024-04-26T20:10:13.930305+0000","last_fullsized":"2024-04-26T20:10:13.930305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:04:37.633017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.769762+0000","last_change":"2024-04-26T20:08:25.703811+0000","last_active":"2024-04-26T20:10:13.769762+0000","last_peered":"2024-04-26T20:10:13.769762+0000","last_clean":"2024-04-26T20:10:13.769762+0000","last_became_active":"2024-04-26T20:08:25.703247+0000","last_became_peered":"2024-04-26T20:08:25.703247+0000","last_unstale":"2024-04-26T20:10:13.769762+0000","last_undegraded":"2024-04-26T20:10:13.769762+0000","last_fullsized":"2024-04-26T20:10:13.769762+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:19:19.605775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"94'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767779+0000","last_change":"2024-04-26T20:08:15.628275+0000","last_active":"2024-04-26T20:10:13.767779+0000","last_peered":"2024-04-26T20:10:13.767779+0000","last_clean":"2024-04-26T20:10:13.767779+0000","last_became_active":"2024-04-26T20:08:15.627115+0000","last_became_peered":"2024-04-26T20:08:15.627115+0000","last_unstale":"2024-04-26T20:10:13.767779+0000","last_undegraded":"2024-04-26T20:10:13.767779+0000","last_fullsized":"2024-04-26T20:10:13.767779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:48.799646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.351023+0000","last_change":"2024-04-26T20:08:17.651666+0000","last_active":"2024-04-26T20:10:14.351023+0000","last_peered":"2024-04-26T20:10:14.351023+0000","last_clean":"2024-04-26T20:10:14.351023+0000","last_became_active":"2024-04-26T20:08:17.644762+0000","last_became_peered":"2024-04-26T20:08:17.644762+0000","last_unstale":"2024-04-26T20:10:14.351023+0000","last_undegraded":"2024-04-26T20:10:14.351023+0000","last_fullsized":"2024-04-26T20:10:14.351023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:19.070044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347175+0000","last_change":"2024-04-26T20:09:13.314326+0000","last_active":"2024-04-26T20:10:14.347175+0000","last_peered":"2024-04-26T20:10:14.347175+0000","last_clean":"2024-04-26T20:10:14.347175+0000","last_became_active":"2024-04-26T20:09:13.313823+0000","last_became_peered":"2024-04-26T20:09:13.313823+0000","last_unstale":"2024-04-26T20:10:14.347175+0000","last_undegraded":"2024-04-26T20:10:14.347175+0000","last_fullsized":"2024-04-26T20:10:14.347175+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:12:55.398462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"94'4","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766361+0000","last_change":"2024-04-26T20:08:16.123015+0000","last_active":"2024-04-26T20:10:13.766361+0000","last_peered":"2024-04-26T20:10:13.766361+0000","last_clean":"2024-04-26T20:10:13.766361+0000","last_became_active":"2024-04-26T20:08:16.122594+0000","last_became_peered":"2024-04-26T20:08:16.122594+0000","last_unstale":"2024-04-26T20:10:13.766361+0000","last_undegraded":"2024-04-26T20:10:13.766361+0000","last_fullsized":"2024-04-26T20:10:13.766361+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:54:11.846029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930053+0000","last_change":"2024-04-26T20:08:17.640882+0000","last_active":"2024-04-26T20:10:13.930053+0000","last_peered":"2024-04-26T20:10:13.930053+0000","last_clean":"2024-04-26T20:10:13.930053+0000","last_became_active":"2024-04-26T20:08:17.640399+0000","last_became_peered":"2024-04-26T20:08:17.640399+0000","last_unstale":"2024-04-26T20:10:13.930053+0000","last_undegraded":"2024-04-26T20:10:13.930053+0000","last_fullsized":"2024-04-26T20:10:13.930053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:05:57.554751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770025+0000","last_change":"2024-04-26T20:08:25.713003+0000","last_active":"2024-04-26T20:10:13.770025+0000","last_peered":"2024-04-26T20:10:13.770025+0000","last_clean":"2024-04-26T20:10:13.770025+0000","last_became_active":"2024-04-26T20:08:25.711116+0000","last_became_peered":"2024-04-26T20:08:25.711116+0000","last_unstale":"2024-04-26T20:10:13.770025+0000","last_undegraded":"2024-04-26T20:10:13.770025+0000","last_fullsized":"2024-04-26T20:10:13.770025+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:45:50.627388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'34","reported_seq":292,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:09.007548+0000","last_change":"2024-04-26T20:08:15.629344+0000","last_active":"2024-04-26T20:11:09.007548+0000","last_peered":"2024-04-26T20:11:09.007548+0000","last_clean":"2024-04-26T20:11:09.007548+0000","last_became_active":"2024-04-26T20:08:15.628836+0000","last_became_peered":"2024-04-26T20:08:15.628836+0000","last_unstale":"2024-04-26T20:11:09.007548+0000","last_undegraded":"2024-04-26T20:11:09.007548+0000","last_fullsized":"2024-04-26T20:11:09.007548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:05:27.011109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57111,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":219,"num_read_kb":165,"num_write":35,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767737+0000","last_change":"2024-04-26T20:08:17.641469+0000","last_active":"2024-04-26T20:10:13.767737+0000","last_peered":"2024-04-26T20:10:13.767737+0000","last_clean":"2024-04-26T20:10:13.767737+0000","last_became_active":"2024-04-26T20:08:17.641147+0000","last_became_peered":"2024-04-26T20:08:17.641147+0000","last_unstale":"2024-04-26T20:10:13.767737+0000","last_undegraded":"2024-04-26T20:10:13.767737+0000","last_fullsized":"2024-04-26T20:10:13.767737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:36:02.885095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766776+0000","last_change":"2024-04-26T20:08:25.708459+0000","last_active":"2024-04-26T20:10:13.766776+0000","last_peered":"2024-04-26T20:10:13.766776+0000","last_clean":"2024-04-26T20:10:13.766776+0000","last_became_active":"2024-04-26T20:08:25.708099+0000","last_became_peered":"2024-04-26T20:08:25.708099+0000","last_unstale":"2024-04-26T20:10:13.766776+0000","last_undegraded":"2024-04-26T20:10:13.766776+0000","last_fullsized":"2024-04-26T20:10:13.766776+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:37.423731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"94'6","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764032+0000","last_change":"2024-04-26T20:08:15.627399+0000","last_active":"2024-04-26T20:10:13.764032+0000","last_peered":"2024-04-26T20:10:13.764032+0000","last_clean":"2024-04-26T20:10:13.764032+0000","last_became_active":"2024-04-26T20:08:15.627018+0000","last_became_peered":"2024-04-26T20:08:15.627018+0000","last_unstale":"2024-04-26T20:10:13.764032+0000","last_undegraded":"2024-04-26T20:10:13.764032+0000","last_fullsized":"2024-04-26T20:10:13.764032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:13.491035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767541+0000","last_change":"2024-04-26T20:08:17.669883+0000","last_active":"2024-04-26T20:10:13.767541+0000","last_peered":"2024-04-26T20:10:13.767541+0000","last_clean":"2024-04-26T20:10:13.767541+0000","last_became_active":"2024-04-26T20:08:17.669484+0000","last_became_peered":"2024-04-26T20:08:17.669484+0000","last_unstale":"2024-04-26T20:10:13.767541+0000","last_undegraded":"2024-04-26T20:10:13.767541+0000","last_fullsized":"2024-04-26T20:10:13.767541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:53:28.820771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765415+0000","last_change":"2024-04-26T20:08:25.694352+0000","last_active":"2024-04-26T20:10:13.765415+0000","last_peered":"2024-04-26T20:10:13.765415+0000","last_clean":"2024-04-26T20:10:13.765415+0000","last_became_active":"2024-04-26T20:08:25.694022+0000","last_became_peered":"2024-04-26T20:08:25.694022+0000","last_unstale":"2024-04-26T20:10:13.765415+0000","last_undegraded":"2024-04-26T20:10:13.765415+0000","last_fullsized":"2024-04-26T20:10:13.765415+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:23.830964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"94'3","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766290+0000","last_change":"2024-04-26T20:08:15.625816+0000","last_active":"2024-04-26T20:10:13.766290+0000","last_peered":"2024-04-26T20:10:13.766290+0000","last_clean":"2024-04-26T20:10:13.766290+0000","last_became_active":"2024-04-26T20:08:15.625390+0000","last_became_peered":"2024-04-26T20:08:15.625390+0000","last_unstale":"2024-04-26T20:10:13.766290+0000","last_undegraded":"2024-04-26T20:10:13.766290+0000","last_fullsized":"2024-04-26T20:10:13.766290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:08:57.864795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767621+0000","last_change":"2024-04-26T20:08:17.653016+0000","last_active":"2024-04-26T20:10:13.767621+0000","last_peered":"2024-04-26T20:10:13.767621+0000","last_clean":"2024-04-26T20:10:13.767621+0000","last_became_active":"2024-04-26T20:08:17.652403+0000","last_became_peered":"2024-04-26T20:08:17.652403+0000","last_unstale":"2024-04-26T20:10:13.767621+0000","last_undegraded":"2024-04-26T20:10:13.767621+0000","last_fullsized":"2024-04-26T20:10:13.767621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:02.580052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770803+0000","last_change":"2024-04-26T20:08:25.717469+0000","last_active":"2024-04-26T20:10:13.770803+0000","last_peered":"2024-04-26T20:10:13.770803+0000","last_clean":"2024-04-26T20:10:13.770803+0000","last_became_active":"2024-04-26T20:08:25.716701+0000","last_became_peered":"2024-04-26T20:08:25.716701+0000","last_unstale":"2024-04-26T20:10:13.770803+0000","last_undegraded":"2024-04-26T20:10:13.770803+0000","last_fullsized":"2024-04-26T20:10:13.770803+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:11:10.991196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'3","reported_seq":148,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:06.767164+0000","last_change":"2024-04-26T20:08:15.626268+0000","last_active":"2024-04-26T20:11:06.767164+0000","last_peered":"2024-04-26T20:11:06.767164+0000","last_clean":"2024-04-26T20:11:06.767164+0000","last_became_active":"2024-04-26T20:08:15.625839+0000","last_became_peered":"2024-04-26T20:08:15.625839+0000","last_unstale":"2024-04-26T20:11:06.767164+0000","last_undegraded":"2024-04-26T20:11:06.767164+0000","last_fullsized":"2024-04-26T20:11:06.767164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:54.537335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767457+0000","last_change":"2024-04-26T20:08:17.644651+0000","last_active":"2024-04-26T20:10:13.767457+0000","last_peered":"2024-04-26T20:10:13.767457+0000","last_clean":"2024-04-26T20:10:13.767457+0000","last_became_active":"2024-04-26T20:08:17.644357+0000","last_became_peered":"2024-04-26T20:08:17.644357+0000","last_unstale":"2024-04-26T20:10:13.767457+0000","last_undegraded":"2024-04-26T20:10:13.767457+0000","last_fullsized":"2024-04-26T20:10:13.767457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:36.344501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768962+0000","last_change":"2024-04-26T20:08:25.710016+0000","last_active":"2024-04-26T20:10:13.768962+0000","last_peered":"2024-04-26T20:10:13.768962+0000","last_clean":"2024-04-26T20:10:13.768962+0000","last_became_active":"2024-04-26T20:08:25.709647+0000","last_became_peered":"2024-04-26T20:08:25.709647+0000","last_unstale":"2024-04-26T20:10:13.768962+0000","last_undegraded":"2024-04-26T20:10:13.768962+0000","last_fullsized":"2024-04-26T20:10:13.768962+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:22.434116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"94'15","reported_seq":55,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768158+0000","last_change":"2024-04-26T20:08:16.128347+0000","last_active":"2024-04-26T20:10:13.768158+0000","last_peered":"2024-04-26T20:10:13.768158+0000","last_clean":"2024-04-26T20:10:13.768158+0000","last_became_active":"2024-04-26T20:08:16.127795+0000","last_became_peered":"2024-04-26T20:08:16.127795+0000","last_unstale":"2024-04-26T20:10:13.768158+0000","last_undegraded":"2024-04-26T20:10:13.768158+0000","last_fullsized":"2024-04-26T20:10:13.768158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:21.217202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770176+0000","last_change":"2024-04-26T20:09:13.314298+0000","last_active":"2024-04-26T20:10:13.770176+0000","last_peered":"2024-04-26T20:10:13.770176+0000","last_clean":"2024-04-26T20:10:13.770176+0000","last_became_active":"2024-04-26T20:09:13.313732+0000","last_became_peered":"2024-04-26T20:09:13.313732+0000","last_unstale":"2024-04-26T20:10:13.770176+0000","last_undegraded":"2024-04-26T20:10:13.770176+0000","last_fullsized":"2024-04-26T20:10:13.770176+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:59:41.782682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765794+0000","last_change":"2024-04-26T20:08:25.700504+0000","last_active":"2024-04-26T20:10:13.765794+0000","last_peered":"2024-04-26T20:10:13.765794+0000","last_clean":"2024-04-26T20:10:13.765794+0000","last_became_active":"2024-04-26T20:08:25.700171+0000","last_became_peered":"2024-04-26T20:08:25.700171+0000","last_unstale":"2024-04-26T20:10:13.765794+0000","last_undegraded":"2024-04-26T20:10:13.765794+0000","last_fullsized":"2024-04-26T20:10:13.765794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:04.442343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'2","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768449+0000","last_change":"2024-04-26T20:08:15.625738+0000","last_active":"2024-04-26T20:10:13.768449+0000","last_peered":"2024-04-26T20:10:13.768449+0000","last_clean":"2024-04-26T20:10:13.768449+0000","last_became_active":"2024-04-26T20:08:15.625381+0000","last_became_peered":"2024-04-26T20:08:15.625381+0000","last_unstale":"2024-04-26T20:10:13.768449+0000","last_undegraded":"2024-04-26T20:10:13.768449+0000","last_fullsized":"2024-04-26T20:10:13.768449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:52.378477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765694+0000","last_change":"2024-04-26T20:08:17.646682+0000","last_active":"2024-04-26T20:10:13.765694+0000","last_peered":"2024-04-26T20:10:13.765694+0000","last_clean":"2024-04-26T20:10:13.765694+0000","last_became_active":"2024-04-26T20:08:17.646362+0000","last_became_peered":"2024-04-26T20:08:17.646362+0000","last_unstale":"2024-04-26T20:10:13.765694+0000","last_undegraded":"2024-04-26T20:10:13.765694+0000","last_fullsized":"2024-04-26T20:10:13.765694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:52.955883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768718+0000","last_change":"2024-04-26T20:08:25.704580+0000","last_active":"2024-04-26T20:10:13.768718+0000","last_peered":"2024-04-26T20:10:13.768718+0000","last_clean":"2024-04-26T20:10:13.768718+0000","last_became_active":"2024-04-26T20:08:25.704083+0000","last_became_peered":"2024-04-26T20:08:25.704083+0000","last_unstale":"2024-04-26T20:10:13.768718+0000","last_undegraded":"2024-04-26T20:10:13.768718+0000","last_fullsized":"2024-04-26T20:10:13.768718+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:59:33.409501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770270+0000","last_change":"2024-04-26T20:08:16.131048+0000","last_active":"2024-04-26T20:10:13.770270+0000","last_peered":"2024-04-26T20:10:13.770270+0000","last_clean":"2024-04-26T20:10:13.770270+0000","last_became_active":"2024-04-26T20:08:16.129700+0000","last_became_peered":"2024-04-26T20:08:16.129700+0000","last_unstale":"2024-04-26T20:10:13.770270+0000","last_undegraded":"2024-04-26T20:10:13.770270+0000","last_fullsized":"2024-04-26T20:10:13.770270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:59:34.814697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763546+0000","last_change":"2024-04-26T20:08:17.669408+0000","last_active":"2024-04-26T20:10:13.763546+0000","last_peered":"2024-04-26T20:10:13.763546+0000","last_clean":"2024-04-26T20:10:13.763546+0000","last_became_active":"2024-04-26T20:08:17.669102+0000","last_became_peered":"2024-04-26T20:08:17.669102+0000","last_unstale":"2024-04-26T20:10:13.763546+0000","last_undegraded":"2024-04-26T20:10:13.763546+0000","last_fullsized":"2024-04-26T20:10:13.763546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:55:29.812387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768743+0000","last_change":"2024-04-26T20:08:25.692910+0000","last_active":"2024-04-26T20:10:13.768743+0000","last_peered":"2024-04-26T20:10:13.768743+0000","last_clean":"2024-04-26T20:10:13.768743+0000","last_became_active":"2024-04-26T20:08:25.692292+0000","last_became_peered":"2024-04-26T20:08:25.692292+0000","last_unstale":"2024-04-26T20:10:13.768743+0000","last_undegraded":"2024-04-26T20:10:13.768743+0000","last_fullsized":"2024-04-26T20:10:13.768743+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:42:19.483756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'1","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768002+0000","last_change":"2024-04-26T20:08:15.633948+0000","last_active":"2024-04-26T20:10:13.768002+0000","last_peered":"2024-04-26T20:10:13.768002+0000","last_clean":"2024-04-26T20:10:13.768002+0000","last_became_active":"2024-04-26T20:08:15.633531+0000","last_became_peered":"2024-04-26T20:08:15.633531+0000","last_unstale":"2024-04-26T20:10:13.768002+0000","last_undegraded":"2024-04-26T20:10:13.768002+0000","last_fullsized":"2024-04-26T20:10:13.768002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:13:15.587856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763403+0000","last_change":"2024-04-26T20:08:17.663674+0000","last_active":"2024-04-26T20:10:13.763403+0000","last_peered":"2024-04-26T20:10:13.763403+0000","last_clean":"2024-04-26T20:10:13.763403+0000","last_became_active":"2024-04-26T20:08:17.663309+0000","last_became_peered":"2024-04-26T20:08:17.663309+0000","last_unstale":"2024-04-26T20:10:13.763403+0000","last_undegraded":"2024-04-26T20:10:13.763403+0000","last_fullsized":"2024-04-26T20:10:13.763403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:55:20.661026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.769298+0000","last_change":"2024-04-26T20:08:25.698807+0000","last_active":"2024-04-26T20:10:13.769298+0000","last_peered":"2024-04-26T20:10:13.769298+0000","last_clean":"2024-04-26T20:10:13.769298+0000","last_became_active":"2024-04-26T20:08:25.697686+0000","last_became_peered":"2024-04-26T20:08:25.697686+0000","last_unstale":"2024-04-26T20:10:13.769298+0000","last_undegraded":"2024-04-26T20:10:13.769298+0000","last_fullsized":"2024-04-26T20:10:13.769298+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:59:37.099531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766429+0000","last_change":"2024-04-26T20:08:17.645055+0000","last_active":"2024-04-26T20:10:13.766429+0000","last_peered":"2024-04-26T20:10:13.766429+0000","last_clean":"2024-04-26T20:10:13.766429+0000","last_became_active":"2024-04-26T20:08:17.644715+0000","last_became_peered":"2024-04-26T20:08:17.644715+0000","last_unstale":"2024-04-26T20:10:13.766429+0000","last_undegraded":"2024-04-26T20:10:13.766429+0000","last_fullsized":"2024-04-26T20:10:13.766429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:12.508925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'4","reported_seq":146,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:11:05.764100+0000","last_change":"2024-04-26T20:08:16.123710+0000","last_active":"2024-04-26T20:11:05.764100+0000","last_peered":"2024-04-26T20:11:05.764100+0000","last_clean":"2024-04-26T20:11:05.764100+0000","last_became_active":"2024-04-26T20:08:16.123046+0000","last_became_peered":"2024-04-26T20:08:16.123046+0000","last_unstale":"2024-04-26T20:11:05.764100+0000","last_undegraded":"2024-04-26T20:11:05.764100+0000","last_fullsized":"2024-04-26T20:11:05.764100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:42.389448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"94'1","reported_seq":27,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767798+0000","last_change":"2024-04-26T20:08:25.698716+0000","last_active":"2024-04-26T20:10:13.767798+0000","last_peered":"2024-04-26T20:10:13.767798+0000","last_clean":"2024-04-26T20:10:13.767798+0000","last_became_active":"2024-04-26T20:08:25.698418+0000","last_became_peered":"2024-04-26T20:08:25.698418+0000","last_unstale":"2024-04-26T20:10:13.767798+0000","last_undegraded":"2024-04-26T20:10:13.767798+0000","last_fullsized":"2024-04-26T20:10:13.767798+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:56.290241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764439+0000","last_change":"2024-04-26T20:08:17.656393+0000","last_active":"2024-04-26T20:10:13.764439+0000","last_peered":"2024-04-26T20:10:13.764439+0000","last_clean":"2024-04-26T20:10:13.764439+0000","last_became_active":"2024-04-26T20:08:17.655914+0000","last_became_peered":"2024-04-26T20:08:17.655914+0000","last_unstale":"2024-04-26T20:10:13.764439+0000","last_undegraded":"2024-04-26T20:10:13.764439+0000","last_fullsized":"2024-04-26T20:10:13.764439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:38:41.459519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"94'3","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347109+0000","last_change":"2024-04-26T20:08:15.631983+0000","last_active":"2024-04-26T20:10:14.347109+0000","last_peered":"2024-04-26T20:10:14.347109+0000","last_clean":"2024-04-26T20:10:14.347109+0000","last_became_active":"2024-04-26T20:08:15.631625+0000","last_became_peered":"2024-04-26T20:08:15.631625+0000","last_unstale":"2024-04-26T20:10:14.347109+0000","last_undegraded":"2024-04-26T20:10:14.347109+0000","last_fullsized":"2024-04-26T20:10:14.347109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:49.264149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":362,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767288+0000","last_change":"2024-04-26T20:06:47.857115+0000","last_active":"2024-04-26T20:10:13.767288+0000","last_peered":"2024-04-26T20:10:13.767288+0000","last_clean":"2024-04-26T20:10:13.767288+0000","last_became_active":"2024-04-26T20:06:47.856652+0000","last_became_peered":"2024-04-26T20:06:47.856652+0000","last_unstale":"2024-04-26T20:10:13.767288+0000","last_undegraded":"2024-04-26T20:10:13.767288+0000","last_fullsized":"2024-04-26T20:10:13.767288+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:02:53.903427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:02:53.903427+0000","last_clean_scrub_stamp":"2024-04-26T20:02:53.903427+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:35:14.359328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768095+0000","last_change":"2024-04-26T20:09:13.283142+0000","last_active":"2024-04-26T20:10:13.768095+0000","last_peered":"2024-04-26T20:10:13.768095+0000","last_clean":"2024-04-26T20:10:13.768095+0000","last_became_active":"2024-04-26T20:09:13.282806+0000","last_became_peered":"2024-04-26T20:09:13.282806+0000","last_unstale":"2024-04-26T20:10:13.768095+0000","last_undegraded":"2024-04-26T20:10:13.768095+0000","last_fullsized":"2024-04-26T20:10:13.768095+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:39.595299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767428+0000","last_change":"2024-04-26T20:08:17.654402+0000","last_active":"2024-04-26T20:10:13.767428+0000","last_peered":"2024-04-26T20:10:13.767428+0000","last_clean":"2024-04-26T20:10:13.767428+0000","last_became_active":"2024-04-26T20:08:17.650047+0000","last_became_peered":"2024-04-26T20:08:17.650047+0000","last_unstale":"2024-04-26T20:10:13.767428+0000","last_undegraded":"2024-04-26T20:10:13.767428+0000","last_fullsized":"2024-04-26T20:10:13.767428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:40.225984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"94'3","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767490+0000","last_change":"2024-04-26T20:08:15.626610+0000","last_active":"2024-04-26T20:10:13.767490+0000","last_peered":"2024-04-26T20:10:13.767490+0000","last_clean":"2024-04-26T20:10:13.767490+0000","last_became_active":"2024-04-26T20:08:15.626162+0000","last_became_peered":"2024-04-26T20:08:15.626162+0000","last_unstale":"2024-04-26T20:10:13.767490+0000","last_undegraded":"2024-04-26T20:10:13.767490+0000","last_fullsized":"2024-04-26T20:10:13.767490+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:20.270714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765912+0000","last_change":"2024-04-26T20:08:25.697123+0000","last_active":"2024-04-26T20:10:13.765912+0000","last_peered":"2024-04-26T20:10:13.765912+0000","last_clean":"2024-04-26T20:10:13.765912+0000","last_became_active":"2024-04-26T20:08:25.696773+0000","last_became_peered":"2024-04-26T20:08:25.696773+0000","last_unstale":"2024-04-26T20:10:13.765912+0000","last_undegraded":"2024-04-26T20:10:13.765912+0000","last_fullsized":"2024-04-26T20:10:13.765912+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:22:56.642427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"94'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767840+0000","last_change":"2024-04-26T20:08:16.128206+0000","last_active":"2024-04-26T20:10:13.767840+0000","last_peered":"2024-04-26T20:10:13.767840+0000","last_clean":"2024-04-26T20:10:13.767840+0000","last_became_active":"2024-04-26T20:08:16.127703+0000","last_became_peered":"2024-04-26T20:08:16.127703+0000","last_unstale":"2024-04-26T20:10:13.767840+0000","last_undegraded":"2024-04-26T20:10:13.767840+0000","last_fullsized":"2024-04-26T20:10:13.767840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:39.852292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768512+0000","last_change":"2024-04-26T20:08:17.652810+0000","last_active":"2024-04-26T20:10:13.768512+0000","last_peered":"2024-04-26T20:10:13.768512+0000","last_clean":"2024-04-26T20:10:13.768512+0000","last_became_active":"2024-04-26T20:08:17.652486+0000","last_became_peered":"2024-04-26T20:08:17.652486+0000","last_unstale":"2024-04-26T20:10:13.768512+0000","last_undegraded":"2024-04-26T20:10:13.768512+0000","last_fullsized":"2024-04-26T20:10:13.768512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:43.382814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764175+0000","last_change":"2024-04-26T20:08:25.701678+0000","last_active":"2024-04-26T20:10:13.764175+0000","last_peered":"2024-04-26T20:10:13.764175+0000","last_clean":"2024-04-26T20:10:13.764175+0000","last_became_active":"2024-04-26T20:08:25.701283+0000","last_became_peered":"2024-04-26T20:08:25.701283+0000","last_unstale":"2024-04-26T20:10:13.764175+0000","last_undegraded":"2024-04-26T20:10:13.764175+0000","last_fullsized":"2024-04-26T20:10:13.764175+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:35:26.774248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"96'11","reported_seq":50,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:27.763015+0000","last_change":"2024-04-26T20:08:15.634215+0000","last_active":"2024-04-26T20:10:27.763015+0000","last_peered":"2024-04-26T20:10:27.763015+0000","last_clean":"2024-04-26T20:10:27.763015+0000","last_became_active":"2024-04-26T20:08:15.633669+0000","last_became_peered":"2024-04-26T20:08:15.633669+0000","last_unstale":"2024-04-26T20:10:27.763015+0000","last_undegraded":"2024-04-26T20:10:27.763015+0000","last_fullsized":"2024-04-26T20:10:27.763015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:06.178974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765223+0000","last_change":"2024-04-26T20:08:17.666557+0000","last_active":"2024-04-26T20:10:13.765223+0000","last_peered":"2024-04-26T20:10:13.765223+0000","last_clean":"2024-04-26T20:10:13.765223+0000","last_became_active":"2024-04-26T20:08:17.665552+0000","last_became_peered":"2024-04-26T20:08:17.665552+0000","last_unstale":"2024-04-26T20:10:13.765223+0000","last_undegraded":"2024-04-26T20:10:13.765223+0000","last_fullsized":"2024-04-26T20:10:13.765223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:11.993087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763576+0000","last_change":"2024-04-26T20:08:25.684541+0000","last_active":"2024-04-26T20:10:13.763576+0000","last_peered":"2024-04-26T20:10:13.763576+0000","last_clean":"2024-04-26T20:10:13.763576+0000","last_became_active":"2024-04-26T20:08:25.684159+0000","last_became_peered":"2024-04-26T20:08:25.684159+0000","last_unstale":"2024-04-26T20:10:13.763576+0000","last_undegraded":"2024-04-26T20:10:13.763576+0000","last_fullsized":"2024-04-26T20:10:13.763576+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:31:45.405767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"94'4","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347923+0000","last_change":"2024-04-26T20:08:16.002624+0000","last_active":"2024-04-26T20:10:14.347923+0000","last_peered":"2024-04-26T20:10:14.347923+0000","last_clean":"2024-04-26T20:10:14.347923+0000","last_became_active":"2024-04-26T20:08:16.002246+0000","last_became_peered":"2024-04-26T20:08:16.002246+0000","last_unstale":"2024-04-26T20:10:14.347923+0000","last_undegraded":"2024-04-26T20:10:14.347923+0000","last_fullsized":"2024-04-26T20:10:14.347923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:59.393586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350768+0000","last_change":"2024-04-26T20:09:13.314961+0000","last_active":"2024-04-26T20:10:14.350768+0000","last_peered":"2024-04-26T20:10:14.350768+0000","last_clean":"2024-04-26T20:10:14.350768+0000","last_became_active":"2024-04-26T20:09:13.314533+0000","last_became_peered":"2024-04-26T20:09:13.314533+0000","last_unstale":"2024-04-26T20:10:14.350768+0000","last_undegraded":"2024-04-26T20:10:14.350768+0000","last_fullsized":"2024-04-26T20:10:14.350768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:29:27.518198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347800+0000","last_change":"2024-04-26T20:08:25.713891+0000","last_active":"2024-04-26T20:10:14.347800+0000","last_peered":"2024-04-26T20:10:14.347800+0000","last_clean":"2024-04-26T20:10:14.347800+0000","last_became_active":"2024-04-26T20:08:25.713480+0000","last_became_peered":"2024-04-26T20:08:25.713480+0000","last_unstale":"2024-04-26T20:10:14.347800+0000","last_undegraded":"2024-04-26T20:10:14.347800+0000","last_fullsized":"2024-04-26T20:10:14.347800+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:26.226811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"94'2","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.929950+0000","last_change":"2024-04-26T20:08:16.123332+0000","last_active":"2024-04-26T20:10:13.929950+0000","last_peered":"2024-04-26T20:10:13.929950+0000","last_clean":"2024-04-26T20:10:13.929950+0000","last_became_active":"2024-04-26T20:08:16.122944+0000","last_became_peered":"2024-04-26T20:08:16.122944+0000","last_unstale":"2024-04-26T20:10:13.929950+0000","last_undegraded":"2024-04-26T20:10:13.929950+0000","last_fullsized":"2024-04-26T20:10:13.929950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:37:23.369867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767503+0000","last_change":"2024-04-26T20:08:17.641693+0000","last_active":"2024-04-26T20:10:13.767503+0000","last_peered":"2024-04-26T20:10:13.767503+0000","last_clean":"2024-04-26T20:10:13.767503+0000","last_became_active":"2024-04-26T20:08:17.641334+0000","last_became_peered":"2024-04-26T20:08:17.641334+0000","last_unstale":"2024-04-26T20:10:13.767503+0000","last_undegraded":"2024-04-26T20:10:13.767503+0000","last_fullsized":"2024-04-26T20:10:13.767503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:34:05.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.770610+0000","last_change":"2024-04-26T20:08:25.714554+0000","last_active":"2024-04-26T20:10:13.770610+0000","last_peered":"2024-04-26T20:10:13.770610+0000","last_clean":"2024-04-26T20:10:13.770610+0000","last_became_active":"2024-04-26T20:08:25.711975+0000","last_became_peered":"2024-04-26T20:08:25.711975+0000","last_unstale":"2024-04-26T20:10:13.770610+0000","last_undegraded":"2024-04-26T20:10:13.770610+0000","last_fullsized":"2024-04-26T20:10:13.770610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:20.061288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"94'3","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764134+0000","last_change":"2024-04-26T20:08:15.634841+0000","last_active":"2024-04-26T20:10:13.764134+0000","last_peered":"2024-04-26T20:10:13.764134+0000","last_clean":"2024-04-26T20:10:13.764134+0000","last_became_active":"2024-04-26T20:08:15.633138+0000","last_became_peered":"2024-04-26T20:08:15.633138+0000","last_unstale":"2024-04-26T20:10:13.764134+0000","last_undegraded":"2024-04-26T20:10:13.764134+0000","last_fullsized":"2024-04-26T20:10:13.764134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:19.364982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765341+0000","last_change":"2024-04-26T20:08:17.650201+0000","last_active":"2024-04-26T20:10:13.765341+0000","last_peered":"2024-04-26T20:10:13.765341+0000","last_clean":"2024-04-26T20:10:13.765341+0000","last_became_active":"2024-04-26T20:08:17.649819+0000","last_became_peered":"2024-04-26T20:08:17.649819+0000","last_unstale":"2024-04-26T20:10:13.765341+0000","last_undegraded":"2024-04-26T20:10:13.765341+0000","last_fullsized":"2024-04-26T20:10:13.765341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:01.772268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"94'1","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347627+0000","last_change":"2024-04-26T20:09:13.315540+0000","last_active":"2024-04-26T20:10:14.347627+0000","last_peered":"2024-04-26T20:10:14.347627+0000","last_clean":"2024-04-26T20:10:14.347627+0000","last_became_active":"2024-04-26T20:09:13.314599+0000","last_became_peered":"2024-04-26T20:09:13.314599+0000","last_unstale":"2024-04-26T20:10:14.347627+0000","last_undegraded":"2024-04-26T20:10:14.347627+0000","last_fullsized":"2024-04-26T20:10:14.347627+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:29.926505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'1","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763660+0000","last_change":"2024-04-26T20:08:15.625290+0000","last_active":"2024-04-26T20:10:13.763660+0000","last_peered":"2024-04-26T20:10:13.763660+0000","last_clean":"2024-04-26T20:10:13.763660+0000","last_became_active":"2024-04-26T20:08:15.624310+0000","last_became_peered":"2024-04-26T20:08:15.624310+0000","last_unstale":"2024-04-26T20:10:13.763660+0000","last_undegraded":"2024-04-26T20:10:13.763660+0000","last_fullsized":"2024-04-26T20:10:13.763660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:04:16.736552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764599+0000","last_change":"2024-04-26T20:08:17.665648+0000","last_active":"2024-04-26T20:10:13.764599+0000","last_peered":"2024-04-26T20:10:13.764599+0000","last_clean":"2024-04-26T20:10:13.764599+0000","last_became_active":"2024-04-26T20:08:17.665205+0000","last_became_peered":"2024-04-26T20:08:17.665205+0000","last_unstale":"2024-04-26T20:10:13.764599+0000","last_undegraded":"2024-04-26T20:10:13.764599+0000","last_fullsized":"2024-04-26T20:10:13.764599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:51.138181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.347741+0000","last_change":"2024-04-26T20:08:25.708815+0000","last_active":"2024-04-26T20:10:14.347741+0000","last_peered":"2024-04-26T20:10:14.347741+0000","last_clean":"2024-04-26T20:10:14.347741+0000","last_became_active":"2024-04-26T20:08:25.708332+0000","last_became_peered":"2024-04-26T20:08:25.708332+0000","last_unstale":"2024-04-26T20:10:14.347741+0000","last_undegraded":"2024-04-26T20:10:14.347741+0000","last_fullsized":"2024-04-26T20:10:14.347741+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:23:19.378956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"94'4","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.773587+0000","last_change":"2024-04-26T20:10:13.773165+0000","last_active":"2024-04-26T20:10:13.773587+0000","last_peered":"2024-04-26T20:10:13.773587+0000","last_clean":"2024-04-26T20:10:13.773587+0000","last_became_active":"2024-04-26T20:10:13.768588+0000","last_became_peered":"2024-04-26T20:10:13.768588+0000","last_unstale":"2024-04-26T20:10:13.773587+0000","last_undegraded":"2024-04-26T20:10:13.773587+0000","last_fullsized":"2024-04-26T20:10:13.773587+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:15.035075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767991+0000","last_change":"2024-04-26T20:08:17.658650+0000","last_active":"2024-04-26T20:10:13.767991+0000","last_peered":"2024-04-26T20:10:13.767991+0000","last_clean":"2024-04-26T20:10:13.767991+0000","last_became_active":"2024-04-26T20:08:17.658325+0000","last_became_peered":"2024-04-26T20:08:17.658325+0000","last_unstale":"2024-04-26T20:10:13.767991+0000","last_undegraded":"2024-04-26T20:10:13.767991+0000","last_fullsized":"2024-04-26T20:10:13.767991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:50:27.167289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.929885+0000","last_change":"2024-04-26T20:08:25.681040+0000","last_active":"2024-04-26T20:10:13.929885+0000","last_peered":"2024-04-26T20:10:13.929885+0000","last_clean":"2024-04-26T20:10:13.929885+0000","last_became_active":"2024-04-26T20:08:25.680632+0000","last_became_peered":"2024-04-26T20:08:25.680632+0000","last_unstale":"2024-04-26T20:10:13.929885+0000","last_undegraded":"2024-04-26T20:10:13.929885+0000","last_fullsized":"2024-04-26T20:10:13.929885+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:16.017414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'3","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764076+0000","last_change":"2024-04-26T20:08:16.123274+0000","last_active":"2024-04-26T20:10:13.764076+0000","last_peered":"2024-04-26T20:10:13.764076+0000","last_clean":"2024-04-26T20:10:13.764076+0000","last_became_active":"2024-04-26T20:08:16.122804+0000","last_became_peered":"2024-04-26T20:08:16.122804+0000","last_unstale":"2024-04-26T20:10:13.764076+0000","last_undegraded":"2024-04-26T20:10:13.764076+0000","last_fullsized":"2024-04-26T20:10:13.764076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:25.235296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767990+0000","last_change":"2024-04-26T20:08:17.641023+0000","last_active":"2024-04-26T20:10:13.767990+0000","last_peered":"2024-04-26T20:10:13.767990+0000","last_clean":"2024-04-26T20:10:13.767990+0000","last_became_active":"2024-04-26T20:08:17.640679+0000","last_became_peered":"2024-04-26T20:08:17.640679+0000","last_unstale":"2024-04-26T20:10:13.767990+0000","last_undegraded":"2024-04-26T20:10:13.767990+0000","last_fullsized":"2024-04-26T20:10:13.767990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:16:37.898958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350108+0000","last_change":"2024-04-26T20:08:25.704251+0000","last_active":"2024-04-26T20:10:14.350108+0000","last_peered":"2024-04-26T20:10:14.350108+0000","last_clean":"2024-04-26T20:10:14.350108+0000","last_became_active":"2024-04-26T20:08:25.702249+0000","last_became_peered":"2024-04-26T20:08:25.702249+0000","last_unstale":"2024-04-26T20:10:14.350108+0000","last_undegraded":"2024-04-26T20:10:14.350108+0000","last_fullsized":"2024-04-26T20:10:14.350108+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:03:12.107793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766554+0000","last_change":"2024-04-26T20:08:15.636474+0000","last_active":"2024-04-26T20:10:13.766554+0000","last_peered":"2024-04-26T20:10:13.766554+0000","last_clean":"2024-04-26T20:10:13.766554+0000","last_became_active":"2024-04-26T20:08:15.636104+0000","last_became_peered":"2024-04-26T20:08:15.636104+0000","last_unstale":"2024-04-26T20:10:13.766554+0000","last_undegraded":"2024-04-26T20:10:13.766554+0000","last_fullsized":"2024-04-26T20:10:13.766554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:09.514033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767700+0000","last_change":"2024-04-26T20:08:17.665683+0000","last_active":"2024-04-26T20:10:13.767700+0000","last_peered":"2024-04-26T20:10:13.767700+0000","last_clean":"2024-04-26T20:10:13.767700+0000","last_became_active":"2024-04-26T20:08:17.665385+0000","last_became_peered":"2024-04-26T20:08:17.665385+0000","last_unstale":"2024-04-26T20:10:13.767700+0000","last_undegraded":"2024-04-26T20:10:13.767700+0000","last_fullsized":"2024-04-26T20:10:13.767700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:00.826582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765673+0000","last_change":"2024-04-26T20:08:25.683715+0000","last_active":"2024-04-26T20:10:13.765673+0000","last_peered":"2024-04-26T20:10:13.765673+0000","last_clean":"2024-04-26T20:10:13.765673+0000","last_became_active":"2024-04-26T20:08:25.683155+0000","last_became_peered":"2024-04-26T20:08:25.683155+0000","last_unstale":"2024-04-26T20:10:13.765673+0000","last_undegraded":"2024-04-26T20:10:13.765673+0000","last_fullsized":"2024-04-26T20:10:13.765673+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:24.656220+0000","last_clean_scrub_stamp":"2024-04-26T20:08:24.656220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:56:44.751519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":51,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.356063+0000","last_change":"2024-04-26T20:10:14.355138+0000","last_active":"2024-04-26T20:10:14.356063+0000","last_peered":"2024-04-26T20:10:14.356063+0000","last_clean":"2024-04-26T20:10:14.356063+0000","last_became_active":"2024-04-26T20:10:14.349682+0000","last_became_peered":"2024-04-26T20:10:14.349682+0000","last_unstale":"2024-04-26T20:10:14.356063+0000","last_undegraded":"2024-04-26T20:10:14.356063+0000","last_fullsized":"2024-04-26T20:10:14.356063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:14.605858+0000","last_clean_scrub_stamp":"2024-04-26T20:08:14.605858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:46:56.365062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767978+0000","last_change":"2024-04-26T20:08:17.642480+0000","last_active":"2024-04-26T20:10:13.767978+0000","last_peered":"2024-04-26T20:10:13.767978+0000","last_clean":"2024-04-26T20:10:13.767978+0000","last_became_active":"2024-04-26T20:08:17.642165+0000","last_became_peered":"2024-04-26T20:08:17.642165+0000","last_unstale":"2024-04-26T20:10:13.767978+0000","last_undegraded":"2024-04-26T20:10:13.767978+0000","last_fullsized":"2024-04-26T20:10:13.767978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:05:39.440740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765537+0000","last_change":"2024-04-26T20:08:17.645823+0000","last_active":"2024-04-26T20:10:13.765537+0000","last_peered":"2024-04-26T20:10:13.765537+0000","last_clean":"2024-04-26T20:10:13.765537+0000","last_became_active":"2024-04-26T20:08:17.645413+0000","last_became_peered":"2024-04-26T20:08:17.645413+0000","last_unstale":"2024-04-26T20:10:13.765537+0000","last_undegraded":"2024-04-26T20:10:13.765537+0000","last_fullsized":"2024-04-26T20:10:13.765537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:36:36.622579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763916+0000","last_change":"2024-04-26T20:08:17.641413+0000","last_active":"2024-04-26T20:10:13.763916+0000","last_peered":"2024-04-26T20:10:13.763916+0000","last_clean":"2024-04-26T20:10:13.763916+0000","last_became_active":"2024-04-26T20:08:17.641073+0000","last_became_peered":"2024-04-26T20:08:17.641073+0000","last_unstale":"2024-04-26T20:10:13.763916+0000","last_undegraded":"2024-04-26T20:10:13.763916+0000","last_fullsized":"2024-04-26T20:10:13.763916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:25:03.961399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.763724+0000","last_change":"2024-04-26T20:08:17.657336+0000","last_active":"2024-04-26T20:10:13.763724+0000","last_peered":"2024-04-26T20:10:13.763724+0000","last_clean":"2024-04-26T20:10:13.763724+0000","last_became_active":"2024-04-26T20:08:17.656982+0000","last_became_peered":"2024-04-26T20:08:17.656982+0000","last_unstale":"2024-04-26T20:10:13.763724+0000","last_undegraded":"2024-04-26T20:10:13.763724+0000","last_fullsized":"2024-04-26T20:10:13.763724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:57:10.149705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:14.350357+0000","last_change":"2024-04-26T20:08:17.662290+0000","last_active":"2024-04-26T20:10:14.350357+0000","last_peered":"2024-04-26T20:10:14.350357+0000","last_clean":"2024-04-26T20:10:14.350357+0000","last_became_active":"2024-04-26T20:08:17.660429+0000","last_became_peered":"2024-04-26T20:08:17.660429+0000","last_unstale":"2024-04-26T20:10:14.350357+0000","last_undegraded":"2024-04-26T20:10:14.350357+0000","last_fullsized":"2024-04-26T20:10:14.350357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:50:03.563066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.768324+0000","last_change":"2024-04-26T20:09:13.284038+0000","last_active":"2024-04-26T20:10:13.768324+0000","last_peered":"2024-04-26T20:10:13.768324+0000","last_clean":"2024-04-26T20:10:13.768324+0000","last_became_active":"2024-04-26T20:09:13.283650+0000","last_became_peered":"2024-04-26T20:09:13.283650+0000","last_unstale":"2024-04-26T20:10:13.768324+0000","last_undegraded":"2024-04-26T20:10:13.768324+0000","last_fullsized":"2024-04-26T20:10:13.768324+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:33.570782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.766369+0000","last_change":"2024-04-26T20:08:17.642813+0000","last_active":"2024-04-26T20:10:13.766369+0000","last_peered":"2024-04-26T20:10:13.766369+0000","last_clean":"2024-04-26T20:10:13.766369+0000","last_became_active":"2024-04-26T20:08:17.642018+0000","last_became_peered":"2024-04-26T20:08:17.642018+0000","last_unstale":"2024-04-26T20:10:13.766369+0000","last_undegraded":"2024-04-26T20:10:13.766369+0000","last_fullsized":"2024-04-26T20:10:13.766369+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:40:19.029810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764452+0000","last_change":"2024-04-26T20:08:17.639030+0000","last_active":"2024-04-26T20:10:13.764452+0000","last_peered":"2024-04-26T20:10:13.764452+0000","last_clean":"2024-04-26T20:10:13.764452+0000","last_became_active":"2024-04-26T20:08:17.638645+0000","last_became_peered":"2024-04-26T20:08:17.638645+0000","last_unstale":"2024-04-26T20:10:13.764452+0000","last_undegraded":"2024-04-26T20:10:13.764452+0000","last_fullsized":"2024-04-26T20:10:13.764452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:58.373648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765398+0000","last_change":"2024-04-26T20:08:17.666873+0000","last_active":"2024-04-26T20:10:13.765398+0000","last_peered":"2024-04-26T20:10:13.765398+0000","last_clean":"2024-04-26T20:10:13.765398+0000","last_became_active":"2024-04-26T20:08:17.666185+0000","last_became_peered":"2024-04-26T20:08:17.666185+0000","last_unstale":"2024-04-26T20:10:13.765398+0000","last_undegraded":"2024-04-26T20:10:13.765398+0000","last_fullsized":"2024-04-26T20:10:13.765398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:51:17.105637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765928+0000","last_change":"2024-04-26T20:08:17.666625+0000","last_active":"2024-04-26T20:10:13.765928+0000","last_peered":"2024-04-26T20:10:13.765928+0000","last_clean":"2024-04-26T20:10:13.765928+0000","last_became_active":"2024-04-26T20:08:17.666288+0000","last_became_peered":"2024-04-26T20:08:17.666288+0000","last_unstale":"2024-04-26T20:10:13.765928+0000","last_undegraded":"2024-04-26T20:10:13.765928+0000","last_fullsized":"2024-04-26T20:10:13.765928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:27.051752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.764134+0000","last_change":"2024-04-26T20:08:17.663490+0000","last_active":"2024-04-26T20:10:13.764134+0000","last_peered":"2024-04-26T20:10:13.764134+0000","last_clean":"2024-04-26T20:10:13.764134+0000","last_became_active":"2024-04-26T20:08:17.663131+0000","last_became_peered":"2024-04-26T20:08:17.663131+0000","last_unstale":"2024-04-26T20:10:13.764134+0000","last_undegraded":"2024-04-26T20:10:13.764134+0000","last_fullsized":"2024-04-26T20:10:13.764134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:46.633558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.930123+0000","last_change":"2024-04-26T20:08:17.664749+0000","last_active":"2024-04-26T20:10:13.930123+0000","last_peered":"2024-04-26T20:10:13.930123+0000","last_clean":"2024-04-26T20:10:13.930123+0000","last_became_active":"2024-04-26T20:08:17.664346+0000","last_became_peered":"2024-04-26T20:08:17.664346+0000","last_unstale":"2024-04-26T20:10:13.930123+0000","last_undegraded":"2024-04-26T20:10:13.930123+0000","last_fullsized":"2024-04-26T20:10:13.930123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:27.106743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.767911+0000","last_change":"2024-04-26T20:08:17.662955+0000","last_active":"2024-04-26T20:10:13.767911+0000","last_peered":"2024-04-26T20:10:13.767911+0000","last_clean":"2024-04-26T20:10:13.767911+0000","last_became_active":"2024-04-26T20:08:17.662639+0000","last_became_peered":"2024-04-26T20:08:17.662639+0000","last_unstale":"2024-04-26T20:10:13.767911+0000","last_undegraded":"2024-04-26T20:10:13.767911+0000","last_fullsized":"2024-04-26T20:10:13.767911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:28.346696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T20:10:13.765592+0000","last_change":"2024-04-26T20:08:17.641465+0000","last_active":"2024-04-26T20:10:13.765592+0000","last_peered":"2024-04-26T20:10:13.765592+0000","last_clean":"2024-04-26T20:10:13.765592+0000","last_became_active":"2024-04-26T20:08:17.641145+0000","last_became_peered":"2024-04-26T20:08:17.641145+0000","last_unstale":"2024-04-26T20:10:13.765592+0000","last_undegraded":"2024-04-26T20:10:13.765592+0000","last_fullsized":"2024-04-26T20:10:13.765592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:08:16.612270+0000","last_clean_scrub_stamp":"2024-04-26T20:08:16.612270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:58.977174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61112,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":646,"num_read_kb":390,"num_write":169,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1595,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383732,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1101824,"data_stored":545440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645367,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28768,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716384,"statfs":{"total":95995035648,"available":95965577216,"internally_reserved":0,"allocated":1785856,"data_stored":1193333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808894,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1114112,"data_stored":545094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070531,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":545014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332169,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708144,"statfs":{"total":95995035648,"available":95957139456,"internally_reserved":0,"allocated":1703936,"data_stored":1140341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495694,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":540270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36480,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708672,"statfs":{"total":95995035648,"available":95957680128,"internally_reserved":0,"allocated":1163264,"data_stored":598304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1118208,"data_stored":541658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411430,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1171456,"data_stored":598019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182496,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":540464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":544920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757333,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28700,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716452,"statfs":{"total":95995035648,"available":95965646848,"internally_reserved":0,"allocated":1716224,"data_stored":1136511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":58124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":57839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T20:11:13.381 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T20:11:13.387 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T20:11:13.393 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T20:11:13.441 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T20:11:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:11:13.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:13 smithi094 ceph-mon[28050]: pgmap v417: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:14.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:13 smithi103 ceph-mon[32635]: pgmap v417: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:14.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:13 smithi204 ceph-mon[32618]: pgmap v417: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.368+0000 7f4216400700 1 -- 172.21.15.94:0/801118273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f421006bac0 msgr2=0x7f4210110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:16.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.368+0000 7f4216400700 1 --2- 172.21.15.94:0/801118273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f421006bac0 0x7f4210110100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f420800b0d0 tx=0x7f420800b3e0 comp rx=0 tx=0).stop 2024-04-26T20:11:16.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.369+0000 7f4216400700 1 -- 172.21.15.94:0/801118273 shutdown_connections 2024-04-26T20:11:16.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.369+0000 7f4216400700 1 --2- 172.21.15.94:0/801118273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f421006bac0 0x7f4210110100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:16.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.369+0000 7f4216400700 1 --2- 172.21.15.94:0/801118273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4210109370 0x7f4210109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:16.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.369+0000 7f4216400700 1 --2- 172.21.15.94:0/801118273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f421006b100 0x7f421006b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:16.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.369+0000 7f4216400700 1 -- 172.21.15.94:0/801118273 >> 172.21.15.94:0/801118273 conn(0x7f42100790d0 msgr2=0x7f42100794e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:16.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.369+0000 7f4216400700 1 -- 172.21.15.94:0/801118273 shutdown_connections 2024-04-26T20:11:16.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.369+0000 7f4216400700 1 -- 172.21.15.94:0/801118273 wait complete. 2024-04-26T20:11:16.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.370+0000 7f4216400700 1 Processor -- start 2024-04-26T20:11:16.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.370+0000 7f4216400700 1 -- start start 2024-04-26T20:11:16.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.370+0000 7f4216400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f421006b100 0x7f42101a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:16.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4216400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421006bac0 0x7f42101a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:16.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4216400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4210109370 0x7f421019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:16.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4216400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4210113470 con 0x7f421006b100 2024-04-26T20:11:16.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4216400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f42101132f0 con 0x7f421006bac0 2024-04-26T20:11:16.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4216400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f42101135f0 con 0x7f4210109370 2024-04-26T20:11:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f420f400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f421006b100 0x7f42101a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4206a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421006bac0 0x7f42101a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:16.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4206a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421006bac0 0x7f42101a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:55122/0 (socket says 172.21.15.94:55122) 2024-04-26T20:11:16.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4206a00700 1 -- 172.21.15.94:0/2040140769 learned_addr learned my addr 172.21.15.94:0/2040140769 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:16.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4214e00700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4210109370 0x7f421019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:16.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4206a00700 1 -- 172.21.15.94:0/2040140769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4210109370 msgr2=0x7f421019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:16.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4206a00700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4210109370 0x7f421019b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:16.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4206a00700 1 -- 172.21.15.94:0/2040140769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f421006b100 msgr2=0x7f42101a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:16.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4206a00700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f421006b100 0x7f42101a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:16.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.371+0000 7f4206a00700 1 -- 172.21.15.94:0/2040140769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421019c030 con 0x7f421006bac0 2024-04-26T20:11:16.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.372+0000 7f4206a00700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421006bac0 0x7f42101a1c50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f41f800e9c0 tx=0x7f41f800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:16.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.372+0000 7f420cc00700 1 -- 172.21.15.94:0/2040140769 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f800cbd0 con 0x7f421006bac0 2024-04-26T20:11:17.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.372+0000 7f420cc00700 1 -- 172.21.15.94:0/2040140769 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f800ee60 con 0x7f421006bac0 2024-04-26T20:11:17.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.372+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421019c320 con 0x7f421006bac0 2024-04-26T20:11:17.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.372+0000 7f420cc00700 1 -- 172.21.15.94:0/2040140769 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f80055c0 con 0x7f421006bac0 2024-04-26T20:11:17.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.372+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42101a85e0 con 0x7f421006bac0 2024-04-26T20:11:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.373+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f421006a110 con 0x7f421006bac0 2024-04-26T20:11:17.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.375+0000 7f420cc00700 1 -- 172.21.15.94:0/2040140769 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f41f8004750 con 0x7f421006bac0 2024-04-26T20:11:17.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.376+0000 7f420cc00700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f41f40a3cd0 0x7f41f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:17.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.377+0000 7f420cc00700 1 -- 172.21.15.94:0/2040140769 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f41f80d0e50 con 0x7f421006bac0 2024-04-26T20:11:17.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.377+0000 7f420f400700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f41f40a3cd0 0x7f41f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:17.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.377+0000 7f420f400700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f41f40a3cd0 0x7f41f40a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f421019ca10 tx=0x7f4200009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:17.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.379+0000 7f420cc00700 1 -- 172.21.15.94:0/2040140769 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f41f8054060 con 0x7f421006bac0 2024-04-26T20:11:17.200 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:16 smithi204 ceph-mon[32618]: pgmap v418: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:17.210 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:16 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2040140769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T20:11:17.253 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:16 smithi103 ceph-mon[32635]: pgmap v418: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:17.315 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:16 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2040140769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T20:11:17.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.634+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f421019ce00 con 0x7f421006bac0 2024-04-26T20:11:17.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.635+0000 7f420cc00700 1 -- 172.21.15.94:0/2040140769 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f421019ce00 con 0x7f421006bac0 2024-04-26T20:11:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:11:17.422 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T20:11:17.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.637+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f41f40a3cd0 msgr2=0x7f41f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:17.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.637+0000 7f4216400700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f41f40a3cd0 0x7f41f40a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f421019ca10 tx=0x7f4200009260 comp rx=0 tx=0).stop 2024-04-26T20:11:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.637+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421006bac0 msgr2=0x7f42101a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:17.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.637+0000 7f4216400700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421006bac0 0x7f42101a1c50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f41f800e9c0 tx=0x7f41f800ecd0 comp rx=0 tx=0).stop 2024-04-26T20:11:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.638+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 shutdown_connections 2024-04-26T20:11:17.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.638+0000 7f4216400700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4210109370 0x7f421019b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:17.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.638+0000 7f4216400700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f421006b100 0x7f42101a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:17.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.638+0000 7f4216400700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f41f40a3cd0 0x7f41f40a6190 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:17.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.638+0000 7f4216400700 1 --2- 172.21.15.94:0/2040140769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f421006bac0 0x7f42101a1c50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:17.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.638+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 >> 172.21.15.94:0/2040140769 conn(0x7f42100790d0 msgr2=0x7f421010e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:17.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.638+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 shutdown_connections 2024-04-26T20:11:17.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:16.638+0000 7f4216400700 1 -- 172.21.15.94:0/2040140769 wait complete. 2024-04-26T20:11:17.736 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:16 smithi094 ceph-mon[28050]: pgmap v418: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:17.748 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:16 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2040140769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T20:11:17.760 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T20:11:17.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:11:18.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.028+0000 7fef95400640 1 -- 172.21.15.94:0/1074195773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef9010ab40 msgr2=0x7fef901113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:18.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.028+0000 7fef95400640 1 --2- 172.21.15.94:0/1074195773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef9010ab40 0x7fef901113d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fef88009be0 tx=0x7fef8802a6c0 comp rx=0 tx=0).stop 2024-04-26T20:11:18.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.028+0000 7fef95400640 1 -- 172.21.15.94:0/1074195773 shutdown_connections 2024-04-26T20:11:18.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.028+0000 7fef95400640 1 --2- 172.21.15.94:0/1074195773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef9010ab40 0x7fef901113d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:18.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.028+0000 7fef95400640 1 --2- 172.21.15.94:0/1074195773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef9010a180 0x7fef9010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:18.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.028+0000 7fef95400640 1 --2- 172.21.15.94:0/1074195773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef90108f90 0x7fef901093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:18.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.028+0000 7fef95400640 1 -- 172.21.15.94:0/1074195773 >> 172.21.15.94:0/1074195773 conn(0x7fef90077f40 msgr2=0x7fef9007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:18.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.028+0000 7fef95400640 1 -- 172.21.15.94:0/1074195773 shutdown_connections 2024-04-26T20:11:18.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.028+0000 7fef95400640 1 -- 172.21.15.94:0/1074195773 wait complete. 2024-04-26T20:11:18.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.029+0000 7fef95400640 1 Processor -- start 2024-04-26T20:11:18.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef95400640 1 -- start start 2024-04-26T20:11:18.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef95400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef90108f90 0x7fef9019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:18.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef95400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef9010a180 0x7fef9019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:18.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef95400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef9010ab40 0x7fef901a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:18.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef95400640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fef90114a90 con 0x7fef9010a180 2024-04-26T20:11:18.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef95400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef90114910 con 0x7fef90108f90 2024-04-26T20:11:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef95400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fef90114c10 con 0x7fef9010ab40 2024-04-26T20:11:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef85e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef9010a180 0x7fef9019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:18.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef85e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef9010a180 0x7fef9019f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56960/0 (socket says 172.21.15.94:56960) 2024-04-26T20:11:18.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef85e00640 1 -- 172.21.15.94:0/3917284939 learned_addr learned my addr 172.21.15.94:0/3917284939 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:18.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.030+0000 7fef8e800640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef90108f90 0x7fef9019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:18.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef8f200640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef9010ab40 0x7fef901a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:18.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef85e00640 1 -- 172.21.15.94:0/3917284939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef9010ab40 msgr2=0x7fef901a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:18.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef85e00640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef9010ab40 0x7fef901a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef85e00640 1 -- 172.21.15.94:0/3917284939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef90108f90 msgr2=0x7fef9019f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:18.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef85e00640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef90108f90 0x7fef9019f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:18.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef85e00640 1 -- 172.21.15.94:0/3917284939 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef901a6cf0 con 0x7fef9010a180 2024-04-26T20:11:18.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef8e800640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef90108f90 0x7fef9019f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:18.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef85e00640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef9010a180 0x7fef9019f610 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fef7800e9c0 tx=0x7fef7800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:18.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef87e00640 1 -- 172.21.15.94:0/3917284939 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7800ccc0 con 0x7fef9010a180 2024-04-26T20:11:18.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.031+0000 7fef87e00640 1 -- 172.21.15.94:0/3917284939 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef780054d0 con 0x7fef9010a180 2024-04-26T20:11:18.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.032+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef901a6fe0 con 0x7fef9010a180 2024-04-26T20:11:18.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.032+0000 7fef87e00640 1 -- 172.21.15.94:0/3917284939 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78010430 con 0x7fef9010a180 2024-04-26T20:11:18.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.032+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef901a74a0 con 0x7fef9010a180 2024-04-26T20:11:18.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.033+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef90110330 con 0x7fef9010a180 2024-04-26T20:11:18.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.035+0000 7fef87e00640 1 -- 172.21.15.94:0/3917284939 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fef78004f80 con 0x7fef9010a180 2024-04-26T20:11:18.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.036+0000 7fef87e00640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fef740a3c40 0x7fef740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:18.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.036+0000 7fef87e00640 1 -- 172.21.15.94:0/3917284939 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fef780d0d60 con 0x7fef9010a180 2024-04-26T20:11:18.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.037+0000 7fef8e800640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fef740a3c40 0x7fef740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:18.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.037+0000 7fef8e800640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fef740a3c40 0x7fef740a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fef90109fe0 tx=0x7fef8000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:19.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.041+0000 7fef87e00640 1 -- 172.21.15.94:0/3917284939 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fef7809d1b0 con 0x7fef9010a180 2024-04-26T20:11:19.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.245+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef9002dd20 con 0x7fef9010a180 2024-04-26T20:11:19.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.249+0000 7fef87e00640 1 -- 172.21.15.94:0/3917284939 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24879 (secure 0 0 0) 0x7fef7809cab0 con 0x7fef9010a180 2024-04-26T20:11:19.033 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:11:19.117 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:10:13.755106+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:11:19.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fef740a3c40 msgr2=0x7fef740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:19.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fef740a3c40 0x7fef740a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fef90109fe0 tx=0x7fef8000b390 comp rx=0 tx=0).stop 2024-04-26T20:11:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef9010a180 msgr2=0x7fef9019f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:19.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef9010a180 0x7fef9019f610 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fef7800e9c0 tx=0x7fef7800ee80 comp rx=0 tx=0).stop 2024-04-26T20:11:19.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 shutdown_connections 2024-04-26T20:11:19.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fef740a3c40 0x7fef740a6100 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:19.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef9010ab40 0x7fef901a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:19.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef9010a180 0x7fef9019f610 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:19.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 --2- 172.21.15.94:0/3917284939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef90108f90 0x7fef9019f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:19.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 >> 172.21.15.94:0/3917284939 conn(0x7fef90077f40 msgr2=0x7fef9010d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:19.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 shutdown_connections 2024-04-26T20:11:19.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:18.252+0000 7fef95400640 1 -- 172.21.15.94:0/3917284939 wait complete. 2024-04-26T20:11:19.448 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:08:13.904332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:08:15.670247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:08:23.682553+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:11:19.462 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:11:19.472 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:18 smithi094 ceph-mon[28050]: pgmap v419: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:19.483 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:18 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3917284939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:11:19.556 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:18 smithi204 ceph-mon[32618]: pgmap v419: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:19.566 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:18 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3917284939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:11:19.601 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:18 smithi103 ceph-mon[32635]: pgmap v419: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:19.656 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:18 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3917284939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:11:19.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.769+0000 7f3da7e00640 1 -- 172.21.15.94:0/4124325329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3da0104b90 msgr2=0x7f3da0104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:19.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.769+0000 7f3da7e00640 1 --2- 172.21.15.94:0/4124325329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3da0104b90 0x7f3da0104fb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009900 tx=0x7f3d9402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:11:19.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.769+0000 7f3da7e00640 1 -- 172.21.15.94:0/4124325329 shutdown_connections 2024-04-26T20:11:19.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.769+0000 7f3da7e00640 1 --2- 172.21.15.94:0/4124325329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3da0106740 0x7f3da010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:19.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.769+0000 7f3da7e00640 1 --2- 172.21.15.94:0/4124325329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3da0105d80 0x7f3da0106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:19.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.769+0000 7f3da7e00640 1 --2- 172.21.15.94:0/4124325329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3da0104b90 0x7f3da0104fb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:19.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.769+0000 7f3da7e00640 1 -- 172.21.15.94:0/4124325329 >> 172.21.15.94:0/4124325329 conn(0x7f3da01000e0 msgr2=0x7f3da0102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:19.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.769+0000 7f3da7e00640 1 -- 172.21.15.94:0/4124325329 shutdown_connections 2024-04-26T20:11:19.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.769+0000 7f3da7e00640 1 -- 172.21.15.94:0/4124325329 wait complete. 2024-04-26T20:11:19.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.770+0000 7f3da7e00640 1 Processor -- start 2024-04-26T20:11:19.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.770+0000 7f3da7e00640 1 -- start start 2024-04-26T20:11:20.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.770+0000 7f3da7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3da0104b90 0x7f3da019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:20.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.770+0000 7f3da7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3da0105d80 0x7f3da019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:20.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.770+0000 7f3da7e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3da0106740 0x7f3da01a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:20.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.770+0000 7f3da7e00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3da0110700 con 0x7f3da0106740 2024-04-26T20:11:20.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.770+0000 7f3da7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3da0110580 con 0x7f3da0105d80 2024-04-26T20:11:20.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.770+0000 7f3da7e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3da0110880 con 0x7f3da0104b90 2024-04-26T20:11:20.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3da0105d80 0x7f3da019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:20.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da5e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3da0106740 0x7f3da01a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:20.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3da0105d80 0x7f3da019b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:55166/0 (socket says 172.21.15.94:55166) 2024-04-26T20:11:20.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da4a00640 1 -- 172.21.15.94:0/2219982972 learned_addr learned my addr 172.21.15.94:0/2219982972 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:20.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da5400640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3da0104b90 0x7f3da019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:20.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da4a00640 1 -- 172.21.15.94:0/2219982972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3da0104b90 msgr2=0x7f3da019ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:20.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da4a00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3da0104b90 0x7f3da019ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:20.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da4a00640 1 -- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3da0106740 msgr2=0x7f3da01a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:20.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da4a00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3da0106740 0x7f3da01a23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:20.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da4a00640 1 -- 172.21.15.94:0/2219982972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da01a2a40 con 0x7f3da0105d80 2024-04-26T20:11:20.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.771+0000 7f3da5e00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3da0106740 0x7f3da01a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:20.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.772+0000 7f3da4a00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3da0105d80 0x7f3da019b360 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00e9f0 tx=0x7f3d8c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:20.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.772+0000 7f3d9e000640 1 -- 172.21.15.94:0/2219982972 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c00cd80 con 0x7f3da0105d80 2024-04-26T20:11:20.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.772+0000 7f3d9e000640 1 -- 172.21.15.94:0/2219982972 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d8c0054d0 con 0x7f3da0105d80 2024-04-26T20:11:20.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.772+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da01a2d30 con 0x7f3da0105d80 2024-04-26T20:11:20.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.772+0000 7f3d9e000640 1 -- 172.21.15.94:0/2219982972 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c010690 con 0x7f3da0105d80 2024-04-26T20:11:20.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.772+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da01a31d0 con 0x7f3da0105d80 2024-04-26T20:11:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.773+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3da010bfd0 con 0x7f3da0105d80 2024-04-26T20:11:20.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.775+0000 7f3d9e000640 1 -- 172.21.15.94:0/2219982972 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3d8c004e90 con 0x7f3da0105d80 2024-04-26T20:11:20.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.777+0000 7f3d9e000640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d880a3c40 0x7f3d880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:20.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.777+0000 7f3d9e000640 1 -- 172.21.15.94:0/2219982972 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f3d8c0d0e50 con 0x7f3da0105d80 2024-04-26T20:11:20.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.777+0000 7f3da5400640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d880a3c40 0x7f3d880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:20.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.778+0000 7f3da5400640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d880a3c40 0x7f3d880a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3d94006010 tx=0x7f3d940314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:20.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.781+0000 7f3d9e000640 1 -- 172.21.15.94:0/2219982972 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3d8c09d2a0 con 0x7f3da0105d80 2024-04-26T20:11:20.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.984+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3da002dd20 con 0x7f3da0105d80 2024-04-26T20:11:20.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.986+0000 7f3d9e000640 1 -- 172.21.15.94:0/2219982972 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f3d8c09cba0 con 0x7f3da0105d80 2024-04-26T20:11:20.779 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:11:20.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d880a3c40 msgr2=0x7f3d880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:20.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d880a3c40 0x7f3d880a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3d94006010 tx=0x7f3d940314a0 comp rx=0 tx=0).stop 2024-04-26T20:11:20.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3da0105d80 msgr2=0x7f3da019b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:20.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3da0105d80 0x7f3da019b360 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c00e9f0 tx=0x7f3d8c00edf0 comp rx=0 tx=0).stop 2024-04-26T20:11:20.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 shutdown_connections 2024-04-26T20:11:20.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3d880a3c40 0x7f3d880a6100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:20.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3da0106740 0x7f3da01a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:21.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3da0105d80 0x7f3da019b360 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:21.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 --2- 172.21.15.94:0/2219982972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3da0104b90 0x7f3da019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:21.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.988+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 >> 172.21.15.94:0/2219982972 conn(0x7f3da01000e0 msgr2=0x7f3da01091a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:21.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.989+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 shutdown_connections 2024-04-26T20:11:21.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:19.989+0000 7f3da7e00640 1 -- 172.21.15.94:0/2219982972 wait complete. 2024-04-26T20:11:21.042 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:11:21.050 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:20 smithi094 ceph-mon[28050]: pgmap v420: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:21.056 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:20 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2219982972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:11:21.133 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:20 smithi103 ceph-mon[32635]: pgmap v420: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:21.212 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:20 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2219982972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:11:21.218 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:20 smithi204 ceph-mon[32618]: pgmap v420: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:21.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:20 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2219982972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:11:21.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.353+0000 7f8e54800640 1 -- 172.21.15.94:0/1823048556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c105d20 msgr2=0x7f8e4c108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.353+0000 7f8e54800640 1 --2- 172.21.15.94:0/1823048556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c105d20 0x7f8e4c108110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009900 tx=0x7f8e3c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:11:21.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.353+0000 7f8e54800640 1 -- 172.21.15.94:0/1823048556 shutdown_connections 2024-04-26T20:11:21.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.353+0000 7f8e54800640 1 --2- 172.21.15.94:0/1823048556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e4c108650 0x7f8e4c10aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:21.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.353+0000 7f8e54800640 1 --2- 172.21.15.94:0/1823048556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c105d20 0x7f8e4c108110 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:21.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.353+0000 7f8e54800640 1 --2- 172.21.15.94:0/1823048556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c1033f0 0x7f8e4c1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:21.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.353+0000 7f8e54800640 1 -- 172.21.15.94:0/1823048556 >> 172.21.15.94:0/1823048556 conn(0x7f8e4c0fcde0 msgr2=0x7f8e4c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:21.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.353+0000 7f8e54800640 1 -- 172.21.15.94:0/1823048556 shutdown_connections 2024-04-26T20:11:21.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.353+0000 7f8e54800640 1 -- 172.21.15.94:0/1823048556 wait complete. 2024-04-26T20:11:21.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.354+0000 7f8e54800640 1 Processor -- start 2024-04-26T20:11:21.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.354+0000 7f8e54800640 1 -- start start 2024-04-26T20:11:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e54800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c1033f0 0x7f8e4c198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:21.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e54800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e4c105d20 0x7f8e4c199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:21.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e54800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c108650 0x7f8e4c1a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:21.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e54800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10e4e0 con 0x7f8e4c1033f0 2024-04-26T20:11:21.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e54800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10e360 con 0x7f8e4c108650 2024-04-26T20:11:21.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e54800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10e660 con 0x7f8e4c105d20 2024-04-26T20:11:21.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e51e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c1033f0 0x7f8e4c198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:21.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e51e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c1033f0 0x7f8e4c198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58308/0 (socket says 172.21.15.94:58308) 2024-04-26T20:11:21.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e51e00640 1 -- 172.21.15.94:0/4198535756 learned_addr learned my addr 172.21.15.94:0/4198535756 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:21.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e51400640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e4c105d20 0x7f8e4c199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:21.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e52800640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c108650 0x7f8e4c1a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:21.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e51e00640 1 -- 172.21.15.94:0/4198535756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e4c105d20 msgr2=0x7f8e4c199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:21.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e51e00640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e4c105d20 0x7f8e4c199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:21.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e51e00640 1 -- 172.21.15.94:0/4198535756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c108650 msgr2=0x7f8e4c1a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:21.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e51e00640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c108650 0x7f8e4c1a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:21.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.355+0000 7f8e51e00640 1 -- 172.21.15.94:0/4198535756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c1a0800 con 0x7f8e4c1033f0 2024-04-26T20:11:21.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.356+0000 7f8e51e00640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c1033f0 0x7f8e4c198be0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009d50 tx=0x7f8e4400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:21.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.356+0000 7f8e42a00640 1 -- 172.21.15.94:0/4198535756 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44003b70 con 0x7f8e4c1033f0 2024-04-26T20:11:21.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.356+0000 7f8e42a00640 1 -- 172.21.15.94:0/4198535756 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e44003d10 con 0x7f8e4c1033f0 2024-04-26T20:11:21.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.356+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4c1a0af0 con 0x7f8e4c1033f0 2024-04-26T20:11:21.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.356+0000 7f8e42a00640 1 -- 172.21.15.94:0/4198535756 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e440108a0 con 0x7f8e4c1033f0 2024-04-26T20:11:21.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.356+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4c1a1010 con 0x7f8e4c1033f0 2024-04-26T20:11:21.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.357+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e4c0fe8d0 con 0x7f8e4c1033f0 2024-04-26T20:11:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.360+0000 7f8e42a00640 1 -- 172.21.15.94:0/4198535756 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8e44004000 con 0x7f8e4c1033f0 2024-04-26T20:11:21.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.361+0000 7f8e42a00640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e380a3c40 0x7f8e380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:21.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.361+0000 7f8e42a00640 1 -- 172.21.15.94:0/4198535756 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f8e440d0fe0 con 0x7f8e4c1033f0 2024-04-26T20:11:22.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.361+0000 7f8e51400640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e380a3c40 0x7f8e380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.362+0000 7f8e51400640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e380a3c40 0x7f8e380a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c005ab0 tx=0x7f8e3c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:22.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.366+0000 7f8e42a00640 1 -- 172.21.15.94:0/4198535756 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8e4409d490 con 0x7f8e4c1033f0 2024-04-26T20:11:22.115 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:21 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4198535756' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:11:22.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.566+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8e4c02dd10 con 0x7f8e4c1033f0 2024-04-26T20:11:22.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.566+0000 7f8e42a00640 1 -- 172.21.15.94:0/4198535756 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f8e4409cd90 con 0x7f8e4c1033f0 2024-04-26T20:11:22.163 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:11:22.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e380a3c40 msgr2=0x7f8e380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:22.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e380a3c40 0x7f8e380a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c005ab0 tx=0x7f8e3c005a20 comp rx=0 tx=0).stop 2024-04-26T20:11:22.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c1033f0 msgr2=0x7f8e4c198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c1033f0 0x7f8e4c198be0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009d50 tx=0x7f8e4400c6a0 comp rx=0 tx=0).stop 2024-04-26T20:11:22.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 shutdown_connections 2024-04-26T20:11:22.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8e380a3c40 0x7f8e380a6100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:22.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c108650 0x7f8e4c1a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:22.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e4c105d20 0x7f8e4c199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:22.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 --2- 172.21.15.94:0/4198535756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e4c1033f0 0x7f8e4c198be0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:22.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.569+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 >> 172.21.15.94:0/4198535756 conn(0x7f8e4c0fcde0 msgr2=0x7f8e4c103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.570+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 shutdown_connections 2024-04-26T20:11:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:21.570+0000 7f8e54800640 1 -- 172.21.15.94:0/4198535756 wait complete. 2024-04-26T20:11:22.423 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:11:22.469 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:21 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4198535756' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:11:22.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:21 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4198535756' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:11:22.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.687+0000 7f30fb800640 1 -- 172.21.15.94:0/1623611315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f4101b30 msgr2=0x7f30f4101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:22.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.687+0000 7f30fb800640 1 --2- 172.21.15.94:0/1623611315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f4101b30 0x7f30f4101f90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009900 tx=0x7f30e002a410 comp rx=0 tx=0).stop 2024-04-26T20:11:22.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.687+0000 7f30fb800640 1 -- 172.21.15.94:0/1623611315 shutdown_connections 2024-04-26T20:11:22.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.687+0000 7f30fb800640 1 --2- 172.21.15.94:0/1623611315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f41024d0 0x7f30f4102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:22.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.687+0000 7f30fb800640 1 --2- 172.21.15.94:0/1623611315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f4101b30 0x7f30f4101f90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:22.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.687+0000 7f30fb800640 1 --2- 172.21.15.94:0/1623611315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30f41011d0 0x7f30f41015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:22.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.687+0000 7f30fb800640 1 -- 172.21.15.94:0/1623611315 >> 172.21.15.94:0/1623611315 conn(0x7f30f40fcdd0 msgr2=0x7f30f40ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:22.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.687+0000 7f30fb800640 1 -- 172.21.15.94:0/1623611315 shutdown_connections 2024-04-26T20:11:22.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.687+0000 7f30fb800640 1 -- 172.21.15.94:0/1623611315 wait complete. 2024-04-26T20:11:22.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.688+0000 7f30fb800640 1 Processor -- start 2024-04-26T20:11:22.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.688+0000 7f30fb800640 1 -- start start 2024-04-26T20:11:22.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30fb800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f41011d0 0x7f30f419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30fb800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30f4101b30 0x7f30f419b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:23.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30fb800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f41024d0 0x7f30f41a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:23.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30fb800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f30f41106c0 con 0x7f30f41011d0 2024-04-26T20:11:23.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30fb800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30f4110540 con 0x7f30f41024d0 2024-04-26T20:11:23.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30fb800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f30f4110840 con 0x7f30f4101b30 2024-04-26T20:11:23.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30f8e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f41011d0 0x7f30f419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:23.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30f8e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f41011d0 0x7f30f419adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58316/0 (socket says 172.21.15.94:58316) 2024-04-26T20:11:23.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30f8e00640 1 -- 172.21.15.94:0/2698194345 learned_addr learned my addr 172.21.15.94:0/2698194345 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:23.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30f3e00640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30f4101b30 0x7f30f419b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:23.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.689+0000 7f30f9800640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f41024d0 0x7f30f41a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:23.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f8e00640 1 -- 172.21.15.94:0/2698194345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30f4101b30 msgr2=0x7f30f419b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:23.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f8e00640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30f4101b30 0x7f30f419b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:23.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f8e00640 1 -- 172.21.15.94:0/2698194345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f41024d0 msgr2=0x7f30f41a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:23.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f8e00640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f41024d0 0x7f30f41a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:23.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f8e00640 1 -- 172.21.15.94:0/2698194345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f41a29c0 con 0x7f30f41011d0 2024-04-26T20:11:23.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f9800640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f41024d0 0x7f30f41a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:11:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f8e00640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f41011d0 0x7f30f419adb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f30e800e9f0 tx=0x7f30e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:23.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f1600640 1 -- 172.21.15.94:0/2698194345 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e800cd80 con 0x7f30f41011d0 2024-04-26T20:11:23.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f1600640 1 -- 172.21.15.94:0/2698194345 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30e80054d0 con 0x7f30f41011d0 2024-04-26T20:11:23.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f41a2cb0 con 0x7f30f41011d0 2024-04-26T20:11:23.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30f1600640 1 -- 172.21.15.94:0/2698194345 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8010690 con 0x7f30f41011d0 2024-04-26T20:11:23.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.690+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f41a31b0 con 0x7f30f41011d0 2024-04-26T20:11:23.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.691+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30f410c120 con 0x7f30f41011d0 2024-04-26T20:11:23.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.693+0000 7f30f1600640 1 -- 172.21.15.94:0/2698194345 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f30e8004e90 con 0x7f30f41011d0 2024-04-26T20:11:23.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.695+0000 7f30f1600640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f30dc0a3c40 0x7f30dc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:23.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.695+0000 7f30f1600640 1 -- 172.21.15.94:0/2698194345 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f30e80d0e80 con 0x7f30f41011d0 2024-04-26T20:11:23.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.695+0000 7f30f3e00640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f30dc0a3c40 0x7f30dc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:23.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.696+0000 7f30f3e00640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f30dc0a3c40 0x7f30dc0a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30e0031500 tx=0x7f30e0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:23.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.699+0000 7f30f1600640 1 -- 172.21.15.94:0/2698194345 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f30e809d330 con 0x7f30f41011d0 2024-04-26T20:11:23.516 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:22 smithi204 ceph-mon[32618]: pgmap v421: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:23.523 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:22 smithi103 ceph-mon[32635]: pgmap v421: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:23.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.901+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f30f402dd60 con 0x7f30f41011d0 2024-04-26T20:11:23.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.902+0000 7f30f1600640 1 -- 172.21.15.94:0/2698194345 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f30e809cc30 con 0x7f30f41011d0 2024-04-26T20:11:23.549 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-26T20:11:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f30dc0a3c40 msgr2=0x7f30dc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:23.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f30dc0a3c40 0x7f30dc0a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30e0031500 tx=0x7f30e0005a20 comp rx=0 tx=0).stop 2024-04-26T20:11:23.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f41011d0 msgr2=0x7f30f419adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:23.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f41011d0 0x7f30f419adb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f30e800e9f0 tx=0x7f30e800edf0 comp rx=0 tx=0).stop 2024-04-26T20:11:23.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 shutdown_connections 2024-04-26T20:11:23.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f30dc0a3c40 0x7f30dc0a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:23.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30f41024d0 0x7f30f41a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:23.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30f4101b30 0x7f30f419b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 --2- 172.21.15.94:0/2698194345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30f41011d0 0x7f30f419adb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:23.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 >> 172.21.15.94:0/2698194345 conn(0x7f30f40fcdd0 msgr2=0x7f30f40fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:23.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.905+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 shutdown_connections 2024-04-26T20:11:23.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:22.906+0000 7f30fb800640 1 -- 172.21.15.94:0/2698194345 wait complete. 2024-04-26T20:11:23.869 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:22 smithi094 ceph-mon[28050]: pgmap v421: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:23.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:11:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2698194345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:11:24.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2698194345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:11:24.056 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2698194345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:11:24.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.105+0000 7f591b200640 1 -- 172.21.15.94:0/2103255187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c106620 msgr2=0x7f591c10ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:24.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.105+0000 7f591b200640 1 --2- 172.21.15.94:0/2103255187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c106620 0x7f591c10ceb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5914009860 tx=0x7f591402a340 comp rx=0 tx=0).stop 2024-04-26T20:11:24.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.106+0000 7f591b200640 1 -- 172.21.15.94:0/2103255187 shutdown_connections 2024-04-26T20:11:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.106+0000 7f591b200640 1 --2- 172.21.15.94:0/2103255187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c106620 0x7f591c10ceb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.106+0000 7f591b200640 1 --2- 172.21.15.94:0/2103255187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c105c60 0x7f591c1060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:24.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.106+0000 7f591b200640 1 --2- 172.21.15.94:0/2103255187 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591c104a70 0x7f591c104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:24.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.106+0000 7f591b200640 1 -- 172.21.15.94:0/2103255187 >> 172.21.15.94:0/2103255187 conn(0x7f591c0fffe0 msgr2=0x7f591c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:24.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.106+0000 7f591b200640 1 -- 172.21.15.94:0/2103255187 shutdown_connections 2024-04-26T20:11:24.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.106+0000 7f591b200640 1 -- 172.21.15.94:0/2103255187 wait complete. 2024-04-26T20:11:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f591b200640 1 Processor -- start 2024-04-26T20:11:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f591b200640 1 -- start start 2024-04-26T20:11:24.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f591b200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c104a70 0x7f591c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:24.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f591b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591c105c60 0x7f591c19b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:24.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f591b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c106620 0x7f591c1a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:24.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f591b200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f591c1105a0 con 0x7f591c104a70 2024-04-26T20:11:24.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f591b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f591c110420 con 0x7f591c106620 2024-04-26T20:11:24.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f591b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f591c110720 con 0x7f591c105c60 2024-04-26T20:11:24.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f5919e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c104a70 0x7f591c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:24.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.107+0000 7f5919400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591c105c60 0x7f591c19b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:24.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c104a70 0x7f591c19abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58328/0 (socket says 172.21.15.94:58328) 2024-04-26T20:11:24.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591c105c60 0x7f591c19b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:46742/0 (socket says 172.21.15.94:46742) 2024-04-26T20:11:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919e00640 1 -- 172.21.15.94:0/3549866390 learned_addr learned my addr 172.21.15.94:0/3549866390 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:24.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f591a800640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c106620 0x7f591c1a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:24.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919400640 1 -- 172.21.15.94:0/3549866390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c106620 msgr2=0x7f591c1a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:24.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919400640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c106620 0x7f591c1a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:24.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919400640 1 -- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c104a70 msgr2=0x7f591c19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:24.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919400640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c104a70 0x7f591c19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:24.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919400640 1 -- 172.21.15.94:0/3549866390 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591c1a2810 con 0x7f591c105c60 2024-04-26T20:11:24.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919e00640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c104a70 0x7f591c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:24.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.108+0000 7f5919400640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591c105c60 0x7f591c19b130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5904005fd0 tx=0x7f590400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:24.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.109+0000 7f5912a00640 1 -- 172.21.15.94:0/3549866390 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590400cd00 con 0x7f591c105c60 2024-04-26T20:11:24.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.109+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591c1a2b00 con 0x7f591c105c60 2024-04-26T20:11:24.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.109+0000 7f5912a00640 1 -- 172.21.15.94:0/3549866390 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59040035f0 con 0x7f591c105c60 2024-04-26T20:11:24.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.109+0000 7f5912a00640 1 -- 172.21.15.94:0/3549866390 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904010690 con 0x7f591c105c60 2024-04-26T20:11:24.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.109+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591c1a3050 con 0x7f591c105c60 2024-04-26T20:11:24.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.110+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f591c10bd90 con 0x7f591c105c60 2024-04-26T20:11:24.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.113+0000 7f5912a00640 1 -- 172.21.15.94:0/3549866390 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5904003a80 con 0x7f591c105c60 2024-04-26T20:11:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.114+0000 7f5912a00640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f59000a3c40 0x7f59000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:24.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.114+0000 7f5912a00640 1 -- 172.21.15.94:0/3549866390 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f59040d0c90 con 0x7f591c105c60 2024-04-26T20:11:24.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.114+0000 7f5919e00640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f59000a3c40 0x7f59000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:24.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.115+0000 7f5919e00640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f59000a3c40 0x7f59000a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f590c005f40 tx=0x7f590c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:24.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.116+0000 7f5912a00640 1 -- 172.21.15.94:0/3549866390 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5904009510 con 0x7f591c105c60 2024-04-26T20:11:24.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.314+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f591c02dc60 con 0x7f591c105c60 2024-04-26T20:11:24.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.315+0000 7f5912a00640 1 -- 172.21.15.94:0/3549866390 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f590409d140 con 0x7f591c105c60 2024-04-26T20:11:25.023 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:11:25.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f59000a3c40 msgr2=0x7f59000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:25.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f59000a3c40 0x7f59000a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f590c005f40 tx=0x7f590c009500 comp rx=0 tx=0).stop 2024-04-26T20:11:25.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591c105c60 msgr2=0x7f591c19b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:25.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591c105c60 0x7f591c19b130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5904005fd0 tx=0x7f590400ee50 comp rx=0 tx=0).stop 2024-04-26T20:11:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 shutdown_connections 2024-04-26T20:11:25.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f59000a3c40 0x7f59000a6100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c106620 0x7f591c1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591c105c60 0x7f591c19b130 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 --2- 172.21.15.94:0/3549866390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c104a70 0x7f591c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 >> 172.21.15.94:0/3549866390 conn(0x7f591c0fffe0 msgr2=0x7f591c108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:25.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.318+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 shutdown_connections 2024-04-26T20:11:25.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:24.319+0000 7f591b200640 1 -- 172.21.15.94:0/3549866390 wait complete. 2024-04-26T20:11:25.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:11:25.266 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:24 smithi094 ceph-mon[28050]: pgmap v422: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:25.274 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:24 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3549866390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:11:25.284 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:24 smithi204 ceph-mon[32618]: pgmap v422: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:25.340 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:24 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3549866390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:11:25.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:24 smithi103 ceph-mon[32635]: pgmap v422: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:25.413 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:24 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3549866390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:11:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.488+0000 7fd855a00640 1 -- 172.21.15.94:0/3939458279 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd850108d90 msgr2=0x7fd8501091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:25.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.488+0000 7fd855a00640 1 --2- 172.21.15.94:0/3939458279 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd850108d90 0x7fd8501091b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd840009900 tx=0x7fd84002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.488+0000 7fd855a00640 1 -- 172.21.15.94:0/3939458279 shutdown_connections 2024-04-26T20:11:25.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.488+0000 7fd855a00640 1 --2- 172.21.15.94:0/3939458279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd85010a940 0x7fd8501111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.488+0000 7fd855a00640 1 --2- 172.21.15.94:0/3939458279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd850109f80 0x7fd85010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.488+0000 7fd855a00640 1 --2- 172.21.15.94:0/3939458279 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd850108d90 0x7fd8501091b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.488+0000 7fd855a00640 1 -- 172.21.15.94:0/3939458279 >> 172.21.15.94:0/3939458279 conn(0x7fd850077d30 msgr2=0x7fd85007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:25.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.488+0000 7fd855a00640 1 -- 172.21.15.94:0/3939458279 shutdown_connections 2024-04-26T20:11:25.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.488+0000 7fd855a00640 1 -- 172.21.15.94:0/3939458279 wait complete. 2024-04-26T20:11:25.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.489+0000 7fd855a00640 1 Processor -- start 2024-04-26T20:11:25.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.489+0000 7fd855a00640 1 -- start start 2024-04-26T20:11:25.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.489+0000 7fd855a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd850108d90 0x7fd85019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:25.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.489+0000 7fd855a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd850109f80 0x7fd85019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:25.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.489+0000 7fd855a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd85010a940 0x7fd8501a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:25.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.489+0000 7fd855a00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd8501148b0 con 0x7fd850108d90 2024-04-26T20:11:25.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.489+0000 7fd855a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd850114730 con 0x7fd850109f80 2024-04-26T20:11:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.489+0000 7fd855a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd850114a30 con 0x7fd85010a940 2024-04-26T20:11:25.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd84fe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd850108d90 0x7fd85019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:25.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd847e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd850109f80 0x7fd85019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:25.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd84fe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd850108d90 0x7fd85019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58354/0 (socket says 172.21.15.94:58354) 2024-04-26T20:11:25.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd84fe00640 1 -- 172.21.15.94:0/2815446056 learned_addr learned my addr 172.21.15.94:0/2815446056 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:25.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd855000640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd85010a940 0x7fd8501a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:25.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd84fe00640 1 -- 172.21.15.94:0/2815446056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd85010a940 msgr2=0x7fd8501a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:25.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd84fe00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd85010a940 0x7fd8501a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd84fe00640 1 -- 172.21.15.94:0/2815446056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd850109f80 msgr2=0x7fd85019f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:25.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd84fe00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd850109f80 0x7fd85019f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:25.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd84fe00640 1 -- 172.21.15.94:0/2815446056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8501a6be0 con 0x7fd850108d90 2024-04-26T20:11:25.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.490+0000 7fd84fe00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd850108d90 0x7fd85019f060 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd840006010 tx=0x7fd8400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:26.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.491+0000 7fd84d600640 1 -- 172.21.15.94:0/2815446056 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84003f030 con 0x7fd850108d90 2024-04-26T20:11:26.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.491+0000 7fd84d600640 1 -- 172.21.15.94:0/2815446056 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd840031910 con 0x7fd850108d90 2024-04-26T20:11:26.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.491+0000 7fd84d600640 1 -- 172.21.15.94:0/2815446056 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd840035070 con 0x7fd850108d90 2024-04-26T20:11:26.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.491+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8501a6e70 con 0x7fd850108d90 2024-04-26T20:11:26.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.491+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8501a7280 con 0x7fd850108d90 2024-04-26T20:11:26.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.492+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd850110180 con 0x7fd850108d90 2024-04-26T20:11:26.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.494+0000 7fd84d600640 1 -- 172.21.15.94:0/2815446056 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd840042030 con 0x7fd850108d90 2024-04-26T20:11:26.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.495+0000 7fd84d600640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8340a3c40 0x7fd8340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:26.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.495+0000 7fd84d600640 1 -- 172.21.15.94:0/2815446056 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fd8400f0ed0 con 0x7fd850108d90 2024-04-26T20:11:26.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.496+0000 7fd847e00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8340a3c40 0x7fd8340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:26.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.496+0000 7fd847e00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8340a3c40 0x7fd8340a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd8501a04d0 tx=0x7fd838005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:26.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.500+0000 7fd84d600640 1 -- 172.21.15.94:0/2815446056 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd8400bccd0 con 0x7fd850108d90 2024-04-26T20:11:26.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.735+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd85002dad0 con 0x7fd850108d90 2024-04-26T20:11:26.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.736+0000 7fd84d600640 1 -- 172.21.15.94:0/2815446056 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11878 (secure 0 0 0) 0x7fd8400c0fd0 con 0x7fd850108d90 2024-04-26T20:11:26.298 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:11:26.307 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:09:27.103941+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24562,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24562,24541]},"id":1}]} 2024-04-26T20:11:26.345 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-26T20:11:26.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8340a3c40 msgr2=0x7fd8340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:26.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8340a3c40 0x7fd8340a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd8501a04d0 tx=0x7fd838005f90 comp rx=0 tx=0).stop 2024-04-26T20:11:26.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd850108d90 msgr2=0x7fd85019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:26.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd850108d90 0x7fd85019f060 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd840006010 tx=0x7fd8400042f0 comp rx=0 tx=0).stop 2024-04-26T20:11:26.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 shutdown_connections 2024-04-26T20:11:26.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd8340a3c40 0x7fd8340a6100 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:26.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd85010a940 0x7fd8501a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:26.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd850109f80 0x7fd85019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 --2- 172.21.15.94:0/2815446056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd850108d90 0x7fd85019f060 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:26.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 >> 172.21.15.94:0/2815446056 conn(0x7fd850077d30 msgr2=0x7fd85010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 shutdown_connections 2024-04-26T20:11:26.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:25.740+0000 7fd855a00640 1 -- 172.21.15.94:0/2815446056 wait complete. 2024-04-26T20:11:26.644 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:11:26.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.858+0000 7f33ea000640 1 -- 172.21.15.94:0/3988474996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e4103220 msgr2=0x7f33e4109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:26.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.858+0000 7f33ea000640 1 --2- 172.21.15.94:0/3988474996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e4103220 0x7f33e4109ab0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009ca0 tx=0x7f33dc02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:11:26.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.858+0000 7f33ea000640 1 -- 172.21.15.94:0/3988474996 shutdown_connections 2024-04-26T20:11:26.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.858+0000 7f33ea000640 1 --2- 172.21.15.94:0/3988474996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e4103220 0x7f33e4109ab0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:26.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.858+0000 7f33ea000640 1 --2- 172.21.15.94:0/3988474996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e4102860 0x7f33e4102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:26.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.858+0000 7f33ea000640 1 --2- 172.21.15.94:0/3988474996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e4101670 0x7f33e4101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:26.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.859+0000 7f33ea000640 1 -- 172.21.15.94:0/3988474996 >> 172.21.15.94:0/3988474996 conn(0x7f33e40fcbc0 msgr2=0x7f33e40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:27.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.859+0000 7f33ea000640 1 -- 172.21.15.94:0/3988474996 shutdown_connections 2024-04-26T20:11:27.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.859+0000 7f33ea000640 1 -- 172.21.15.94:0/3988474996 wait complete. 2024-04-26T20:11:27.114 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:26 smithi103 ceph-mon[32635]: pgmap v423: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:27.121 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:26 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2815446056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:11:27.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33ea000640 1 Processor -- start 2024-04-26T20:11:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33ea000640 1 -- start start 2024-04-26T20:11:27.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33ea000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e4101670 0x7f33e419ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:27.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33ea000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e4102860 0x7f33e419b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:27.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33ea000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e4103220 0x7f33e41a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:27.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33ea000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f33e410d1f0 con 0x7f33e4102860 2024-04-26T20:11:27.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33ea000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33e410d070 con 0x7f33e4101670 2024-04-26T20:11:27.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33ea000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f33e410d370 con 0x7f33e4103220 2024-04-26T20:11:27.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33e3e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e4102860 0x7f33e419b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33e3e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e4102860 0x7f33e419b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58372/0 (socket says 172.21.15.94:58372) 2024-04-26T20:11:27.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33e3e00640 1 -- 172.21.15.94:0/1091202952 learned_addr learned my addr 172.21.15.94:0/1091202952 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:27.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33e8c00640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e4101670 0x7f33e419ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:27.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.860+0000 7f33e9600640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e4103220 0x7f33e41a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:27.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33e3e00640 1 -- 172.21.15.94:0/1091202952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e4103220 msgr2=0x7f33e41a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:27.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33e3e00640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e4103220 0x7f33e41a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33e3e00640 1 -- 172.21.15.94:0/1091202952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e4101670 msgr2=0x7f33e419ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:27.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33e3e00640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e4101670 0x7f33e419ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:27.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33e3e00640 1 -- 172.21.15.94:0/1091202952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e41a2720 con 0x7f33e4102860 2024-04-26T20:11:27.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33e3e00640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e4102860 0x7f33e419b040 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f33d000e9f0 tx=0x7f33d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:27.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33e1600640 1 -- 172.21.15.94:0/1091202952 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d000cd80 con 0x7f33e4102860 2024-04-26T20:11:27.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e41a2a10 con 0x7f33e4102860 2024-04-26T20:11:27.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33e1600640 1 -- 172.21.15.94:0/1091202952 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33d00054d0 con 0x7f33e4102860 2024-04-26T20:11:27.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33e1600640 1 -- 172.21.15.94:0/1091202952 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0010690 con 0x7f33e4102860 2024-04-26T20:11:27.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.861+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e41a2f60 con 0x7f33e4102860 2024-04-26T20:11:27.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.862+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e41a31f0 con 0x7f33e4102860 2024-04-26T20:11:27.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.864+0000 7f33e1600640 1 -- 172.21.15.94:0/1091202952 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f33d0004e90 con 0x7f33e4102860 2024-04-26T20:11:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.866+0000 7f33e1600640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33cc0a3c40 0x7f33cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:27.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.866+0000 7f33e1600640 1 -- 172.21.15.94:0/1091202952 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f33d00d0df0 con 0x7f33e4102860 2024-04-26T20:11:27.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.866+0000 7f33e8c00640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33cc0a3c40 0x7f33cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:27.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.867+0000 7f33e8c00640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33cc0a3c40 0x7f33cc0a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f33d80095f0 tx=0x7f33d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:27.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:26.870+0000 7f33e1600640 1 -- 172.21.15.94:0/1091202952 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f33d009d2a0 con 0x7f33e4102860 2024-04-26T20:11:27.659 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:26 smithi094 ceph-mon[28050]: pgmap v423: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:27.665 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:26 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2815446056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:11:27.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.074+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33e41a33c0 con 0x7f33e4102860 2024-04-26T20:11:27.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.077+0000 7f33e1600640 1 -- 172.21.15.94:0/1091202952 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24879 (secure 0 0 0) 0x7f33d009cba0 con 0x7f33e4102860 2024-04-26T20:11:27.728 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:11:27.737 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:10:13.755106+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:11:27.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33cc0a3c40 msgr2=0x7f33cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:27.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33cc0a3c40 0x7f33cc0a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f33d80095f0 tx=0x7f33d8005f90 comp rx=0 tx=0).stop 2024-04-26T20:11:27.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e4102860 msgr2=0x7f33e419b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:27.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e4102860 0x7f33e419b040 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f33d000e9f0 tx=0x7f33d000edf0 comp rx=0 tx=0).stop 2024-04-26T20:11:27.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 shutdown_connections 2024-04-26T20:11:27.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f33cc0a3c40 0x7f33cc0a6100 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:27.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33e4103220 0x7f33e41a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:27.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33e4102860 0x7f33e419b040 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:27.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 --2- 172.21.15.94:0/1091202952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e4101670 0x7f33e419ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:27.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.079+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 >> 172.21.15.94:0/1091202952 conn(0x7f33e40fcbc0 msgr2=0x7f33e40fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:28.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.080+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 shutdown_connections 2024-04-26T20:11:28.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:27.080+0000 7f33ea000640 1 -- 172.21.15.94:0/1091202952 wait complete. 2024-04-26T20:11:28.070 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:08:13.904332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:08:15.670247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:08:23.682553+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:11:28.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:11:28.091 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:27 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1091202952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:11:28.105 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:27 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1091202952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:11:28.118 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:26 smithi204 ceph-mon[32618]: pgmap v423: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:28.160 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:26 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2815446056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:11:28.206 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:27 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1091202952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:11:28.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.328+0000 7fe488800640 1 -- 172.21.15.94:0/3928548713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480106740 msgr2=0x7fe48010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:28.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.328+0000 7fe488800640 1 --2- 172.21.15.94:0/3928548713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480106740 0x7fe48010cfd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009ca0 tx=0x7fe47c02a8b0 comp rx=0 tx=0).stop 2024-04-26T20:11:28.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.328+0000 7fe488800640 1 -- 172.21.15.94:0/3928548713 shutdown_connections 2024-04-26T20:11:28.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.328+0000 7fe488800640 1 --2- 172.21.15.94:0/3928548713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480106740 0x7fe48010cfd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:28.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.328+0000 7fe488800640 1 --2- 172.21.15.94:0/3928548713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe480105d80 0x7fe480106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:28.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.328+0000 7fe488800640 1 --2- 172.21.15.94:0/3928548713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe480104b90 0x7fe480104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:28.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.328+0000 7fe488800640 1 -- 172.21.15.94:0/3928548713 >> 172.21.15.94:0/3928548713 conn(0x7fe4801000e0 msgr2=0x7fe480102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:28.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.328+0000 7fe488800640 1 -- 172.21.15.94:0/3928548713 shutdown_connections 2024-04-26T20:11:28.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.328+0000 7fe488800640 1 -- 172.21.15.94:0/3928548713 wait complete. 2024-04-26T20:11:28.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.329+0000 7fe488800640 1 Processor -- start 2024-04-26T20:11:28.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.329+0000 7fe488800640 1 -- start start 2024-04-26T20:11:28.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.329+0000 7fe488800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480104b90 0x7fe48019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:28.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.329+0000 7fe488800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe480105d80 0x7fe48019b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:28.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.329+0000 7fe488800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe480106740 0x7fe4801a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:28.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.329+0000 7fe488800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe4801106a0 con 0x7fe480104b90 2024-04-26T20:11:28.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.329+0000 7fe488800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe480110520 con 0x7fe480105d80 2024-04-26T20:11:28.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.329+0000 7fe488800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe480110820 con 0x7fe480106740 2024-04-26T20:11:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480104b90 0x7fe48019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:28.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480104b90 0x7fe48019ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58386/0 (socket says 172.21.15.94:58386) 2024-04-26T20:11:28.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485e00640 1 -- 172.21.15.94:0/3674705924 learned_addr learned my addr 172.21.15.94:0/3674705924 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:28.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485400640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe480105d80 0x7fe48019b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe486800640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe480106740 0x7fe4801a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:28.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485e00640 1 -- 172.21.15.94:0/3674705924 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe480106740 msgr2=0x7fe4801a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:28.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485e00640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe480106740 0x7fe4801a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:28.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485e00640 1 -- 172.21.15.94:0/3674705924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe480105d80 msgr2=0x7fe48019b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:28.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485e00640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe480105d80 0x7fe48019b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:28.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485e00640 1 -- 172.21.15.94:0/3674705924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4801a2960 con 0x7fe480104b90 2024-04-26T20:11:28.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.330+0000 7fe485e00640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480104b90 0x7fe48019ad40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe47400e9c0 tx=0x7fe47400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:28.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.331+0000 7fe47aa00640 1 -- 172.21.15.94:0/3674705924 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47400cd80 con 0x7fe480104b90 2024-04-26T20:11:28.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.331+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4801a2c50 con 0x7fe480104b90 2024-04-26T20:11:28.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.331+0000 7fe47aa00640 1 -- 172.21.15.94:0/3674705924 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4740035f0 con 0x7fe480104b90 2024-04-26T20:11:28.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.331+0000 7fe47aa00640 1 -- 172.21.15.94:0/3674705924 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474010690 con 0x7fe480104b90 2024-04-26T20:11:28.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.331+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4801a31a0 con 0x7fe480104b90 2024-04-26T20:11:28.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.332+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe48010beb0 con 0x7fe480104b90 2024-04-26T20:11:28.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.334+0000 7fe47aa00640 1 -- 172.21.15.94:0/3674705924 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe4740037a0 con 0x7fe480104b90 2024-04-26T20:11:28.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.335+0000 7fe47aa00640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe4680a3c40 0x7fe4680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:28.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.335+0000 7fe47aa00640 1 -- 172.21.15.94:0/3674705924 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fe4740d0d20 con 0x7fe480104b90 2024-04-26T20:11:28.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.336+0000 7fe485400640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe4680a3c40 0x7fe4680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:29.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.336+0000 7fe485400640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe4680a3c40 0x7fe4680a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe48019c250 tx=0x7fe470005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:29.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.339+0000 7fe47aa00640 1 -- 172.21.15.94:0/3674705924 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe47409d170 con 0x7fe480104b90 2024-04-26T20:11:29.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.540+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe48002dad0 con 0x7fe480104b90 2024-04-26T20:11:29.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.541+0000 7fe47aa00640 1 -- 172.21.15.94:0/3674705924 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7fe48002dad0 con 0x7fe480104b90 2024-04-26T20:11:29.114 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:11:29.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe4680a3c40 msgr2=0x7fe4680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:29.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe4680a3c40 0x7fe4680a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe48019c250 tx=0x7fe470005f90 comp rx=0 tx=0).stop 2024-04-26T20:11:29.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480104b90 msgr2=0x7fe48019ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:29.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480104b90 0x7fe48019ad40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe47400e9c0 tx=0x7fe47400ee80 comp rx=0 tx=0).stop 2024-04-26T20:11:29.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 shutdown_connections 2024-04-26T20:11:29.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe4680a3c40 0x7fe4680a6100 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe480106740 0x7fe4801a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe480105d80 0x7fe48019b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 --2- 172.21.15.94:0/3674705924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe480104b90 0x7fe48019ad40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 >> 172.21.15.94:0/3674705924 conn(0x7fe4801000e0 msgr2=0x7fe480109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 shutdown_connections 2024-04-26T20:11:29.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:28.544+0000 7fe488800640 1 -- 172.21.15.94:0/3674705924 wait complete. 2024-04-26T20:11:29.308 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:11:29.315 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:28 smithi094 ceph-mon[28050]: pgmap v424: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:29.320 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:28 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3674705924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:11:29.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:28 smithi204 ceph-mon[32618]: pgmap v424: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:29.389 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:28 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3674705924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:11:29.394 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:28 smithi103 ceph-mon[32635]: pgmap v424: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:29.399 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:28 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3674705924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:11:29.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.537+0000 7fbacea00640 1 -- 172.21.15.94:0/282292550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbac8104a70 msgr2=0x7fbac8104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:29.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.537+0000 7fbacea00640 1 --2- 172.21.15.94:0/282292550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbac8104a70 0x7fbac8104e90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009900 tx=0x7fbabc02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.538+0000 7fbacea00640 1 -- 172.21.15.94:0/282292550 shutdown_connections 2024-04-26T20:11:29.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.538+0000 7fbacea00640 1 --2- 172.21.15.94:0/282292550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac8106620 0x7fbac810ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.538+0000 7fbacea00640 1 --2- 172.21.15.94:0/282292550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac8105c60 0x7fbac81060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.538+0000 7fbacea00640 1 --2- 172.21.15.94:0/282292550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbac8104a70 0x7fbac8104e90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.538+0000 7fbacea00640 1 -- 172.21.15.94:0/282292550 >> 172.21.15.94:0/282292550 conn(0x7fbac80fffe0 msgr2=0x7fbac8102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:29.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.538+0000 7fbacea00640 1 -- 172.21.15.94:0/282292550 shutdown_connections 2024-04-26T20:11:29.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.538+0000 7fbacea00640 1 -- 172.21.15.94:0/282292550 wait complete. 2024-04-26T20:11:29.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.539+0000 7fbacea00640 1 Processor -- start 2024-04-26T20:11:29.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.539+0000 7fbacea00640 1 -- start start 2024-04-26T20:11:29.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.539+0000 7fbacea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbac8104a70 0x7fbac819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:29.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.539+0000 7fbacea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac8105c60 0x7fbac819b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:29.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.539+0000 7fbacea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac8106620 0x7fbac81a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:29.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.539+0000 7fbacea00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbac81105c0 con 0x7fbac8105c60 2024-04-26T20:11:29.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.539+0000 7fbacea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbac8110440 con 0x7fbac8104a70 2024-04-26T20:11:29.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.539+0000 7fbacea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbac8110740 con 0x7fbac8106620 2024-04-26T20:11:29.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbaccc00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac8105c60 0x7fbac819b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:29.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbacd600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbac8104a70 0x7fbac819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:29.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbacd600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbac8104a70 0x7fbac819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:49608/0 (socket says 172.21.15.94:49608) 2024-04-26T20:11:29.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbacd600640 1 -- 172.21.15.94:0/945756535 learned_addr learned my addr 172.21.15.94:0/945756535 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:29.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbaccc00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac8105c60 0x7fbac819b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58400/0 (socket says 172.21.15.94:58400) 2024-04-26T20:11:29.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbace000640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac8106620 0x7fbac81a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:29.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbaccc00640 1 -- 172.21.15.94:0/945756535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac8106620 msgr2=0x7fbac81a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:29.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbaccc00640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac8106620 0x7fbac81a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:29.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbaccc00640 1 -- 172.21.15.94:0/945756535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbac8104a70 msgr2=0x7fbac819ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:30.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbaccc00640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbac8104a70 0x7fbac819ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:30.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.540+0000 7fbaccc00640 1 -- 172.21.15.94:0/945756535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac81a28d0 con 0x7fbac8105c60 2024-04-26T20:11:30.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.541+0000 7fbaccc00640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac8105c60 0x7fbac819b1d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbab800e9f0 tx=0x7fbab800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:30.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.541+0000 7fbac6000640 1 -- 172.21.15.94:0/945756535 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab800cd80 con 0x7fbac8105c60 2024-04-26T20:11:30.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.541+0000 7fbac6000640 1 -- 172.21.15.94:0/945756535 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbab80035f0 con 0x7fbac8105c60 2024-04-26T20:11:30.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.541+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac81a2bc0 con 0x7fbac8105c60 2024-04-26T20:11:30.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.541+0000 7fbac6000640 1 -- 172.21.15.94:0/945756535 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab8010670 con 0x7fbac8105c60 2024-04-26T20:11:30.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.541+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac81a3110 con 0x7fbac8105c60 2024-04-26T20:11:30.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.542+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac810be10 con 0x7fbac8105c60 2024-04-26T20:11:30.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.544+0000 7fbac6000640 1 -- 172.21.15.94:0/945756535 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbab80037a0 con 0x7fbac8105c60 2024-04-26T20:11:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.545+0000 7fbac6000640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbab00a3c40 0x7fbab00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.546+0000 7fbac6000640 1 -- 172.21.15.94:0/945756535 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fbab80d0c20 con 0x7fbac8105c60 2024-04-26T20:11:30.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.546+0000 7fbacd600640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbab00a3c40 0x7fbab00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:30.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.547+0000 7fbacd600640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbab00a3c40 0x7fbab00a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbabc006010 tx=0x7fbabc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.552+0000 7fbac6000640 1 -- 172.21.15.94:0/945756535 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbab809d070 con 0x7fbac8105c60 2024-04-26T20:11:30.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.767+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbac802dd20 con 0x7fbac8105c60 2024-04-26T20:11:30.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.768+0000 7fbac6000640 1 -- 172.21.15.94:0/945756535 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7fbab809c970 con 0x7fbac8105c60 2024-04-26T20:11:30.343 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:11:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbab00a3c40 msgr2=0x7fbab00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:30.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbab00a3c40 0x7fbab00a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbabc006010 tx=0x7fbabc005a20 comp rx=0 tx=0).stop 2024-04-26T20:11:30.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac8105c60 msgr2=0x7fbac819b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:30.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac8105c60 0x7fbac819b1d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbab800e9f0 tx=0x7fbab800eeb0 comp rx=0 tx=0).stop 2024-04-26T20:11:30.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 shutdown_connections 2024-04-26T20:11:30.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbab00a3c40 0x7fbab00a6100 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:30.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbac8106620 0x7fbac81a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac8105c60 0x7fbac819b1d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:30.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 --2- 172.21.15.94:0/945756535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbac8104a70 0x7fbac819ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:30.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 >> 172.21.15.94:0/945756535 conn(0x7fbac80fffe0 msgr2=0x7fbac8108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:30.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 shutdown_connections 2024-04-26T20:11:30.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:29.771+0000 7fbacea00640 1 -- 172.21.15.94:0/945756535 wait complete. 2024-04-26T20:11:30.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:11:30.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.872+0000 7fb5c7e00640 1 -- 172.21.15.94:0/4133529668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5c8109f80 msgr2=0x7fb5c810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:30.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.872+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/4133529668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5c8109f80 0x7fb5c810a400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0009860 tx=0x7fb5b002a340 comp rx=0 tx=0).stop 2024-04-26T20:11:30.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.873+0000 7fb5c7e00640 1 -- 172.21.15.94:0/4133529668 shutdown_connections 2024-04-26T20:11:30.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.873+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/4133529668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5c810a940 0x7fb5c81111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:30.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.873+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/4133529668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5c8109f80 0x7fb5c810a400 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:30.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.873+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/4133529668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c8108d90 0x7fb5c81091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:31.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.873+0000 7fb5c7e00640 1 -- 172.21.15.94:0/4133529668 >> 172.21.15.94:0/4133529668 conn(0x7fb5c8077d30 msgr2=0x7fb5c807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:31.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.873+0000 7fb5c7e00640 1 -- 172.21.15.94:0/4133529668 shutdown_connections 2024-04-26T20:11:31.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.873+0000 7fb5c7e00640 1 -- 172.21.15.94:0/4133529668 wait complete. 2024-04-26T20:11:31.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:30 smithi103 ceph-mon[32635]: pgmap v425: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:31.089 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:30 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/945756535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:11:31.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.874+0000 7fb5c7e00640 1 Processor -- start 2024-04-26T20:11:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.874+0000 7fb5c7e00640 1 -- start start 2024-04-26T20:11:31.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.874+0000 7fb5c7e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c8108d90 0x7fb5c819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.874+0000 7fb5c7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5c8109f80 0x7fb5c819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:31.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.874+0000 7fb5c7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5c810a940 0x7fb5c81a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:31.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.874+0000 7fb5c7e00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c81148c0 con 0x7fb5c8108d90 2024-04-26T20:11:31.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.874+0000 7fb5c7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c8114740 con 0x7fb5c8109f80 2024-04-26T20:11:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.874+0000 7fb5c7e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c8114a40 con 0x7fb5c810a940 2024-04-26T20:11:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c6a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c8108d90 0x7fb5c819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:31.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5bfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5c8109f80 0x7fb5c819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:31.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c6a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c8108d90 0x7fb5c819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57906/0 (socket says 172.21.15.94:57906) 2024-04-26T20:11:31.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c6a00640 1 -- 172.21.15.94:0/1078431018 learned_addr learned my addr 172.21.15.94:0/1078431018 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:31.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c7400640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5c810a940 0x7fb5c81a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:31.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c6a00640 1 -- 172.21.15.94:0/1078431018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5c810a940 msgr2=0x7fb5c81a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:31.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c6a00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5c810a940 0x7fb5c81a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:31.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c6a00640 1 -- 172.21.15.94:0/1078431018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5c8109f80 msgr2=0x7fb5c819f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:31.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c6a00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5c8109f80 0x7fb5c819f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:31.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c6a00640 1 -- 172.21.15.94:0/1078431018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c81a6bd0 con 0x7fb5c8108d90 2024-04-26T20:11:31.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.875+0000 7fb5c6a00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c8108d90 0x7fb5c819efb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800e9a0 tx=0x7fb5b800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.876+0000 7fb5bf400640 1 -- 172.21.15.94:0/1078431018 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b800cd80 con 0x7fb5c8108d90 2024-04-26T20:11:31.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.876+0000 7fb5bf400640 1 -- 172.21.15.94:0/1078431018 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5b80108e0 con 0x7fb5c8108d90 2024-04-26T20:11:31.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.876+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c81a6ec0 con 0x7fb5c8108d90 2024-04-26T20:11:31.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.876+0000 7fb5bf400640 1 -- 172.21.15.94:0/1078431018 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8018960 con 0x7fb5c8108d90 2024-04-26T20:11:31.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.876+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c81a7320 con 0x7fb5c8108d90 2024-04-26T20:11:31.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.877+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5c8110180 con 0x7fb5c8108d90 2024-04-26T20:11:31.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.879+0000 7fb5bf400640 1 -- 172.21.15.94:0/1078431018 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb5b8010a90 con 0x7fb5c8108d90 2024-04-26T20:11:31.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.882+0000 7fb5bf400640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5ac0a3c40 0x7fb5ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:31.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.882+0000 7fb5bf400640 1 -- 172.21.15.94:0/1078431018 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fb5b80d0cd0 con 0x7fb5c8108d90 2024-04-26T20:11:31.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.882+0000 7fb5bfe00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5ac0a3c40 0x7fb5ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:31.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.882+0000 7fb5bfe00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5ac0a3c40 0x7fb5ac0a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb5b002a930 tx=0x7fb5b0030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:31.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:30.886+0000 7fb5bf400640 1 -- 172.21.15.94:0/1078431018 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb5b8054060 con 0x7fb5c8108d90 2024-04-26T20:11:31.782 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:30 smithi094 ceph-mon[28050]: pgmap v425: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:31.792 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:30 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/945756535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:11:31.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.087+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb5c8063530 con 0x7fb5c8108d90 2024-04-26T20:11:31.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.088+0000 7fb5bf400640 1 -- 172.21.15.94:0/1078431018 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7fb5b809d120 con 0x7fb5c8108d90 2024-04-26T20:11:31.817 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-26T20:11:31.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5ac0a3c40 msgr2=0x7fb5ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5ac0a3c40 0x7fb5ac0a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb5b002a930 tx=0x7fb5b0030450 comp rx=0 tx=0).stop 2024-04-26T20:11:31.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c8108d90 msgr2=0x7fb5c819efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:31.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c8108d90 0x7fb5c819efb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800e9a0 tx=0x7fb5b800ee60 comp rx=0 tx=0).stop 2024-04-26T20:11:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 shutdown_connections 2024-04-26T20:11:31.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fb5ac0a3c40 0x7fb5ac0a6100 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:31.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5c810a940 0x7fb5c81a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:31.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5c8109f80 0x7fb5c819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:31.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 --2- 172.21.15.94:0/1078431018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb5c8108d90 0x7fb5c819efb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:32.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 >> 172.21.15.94:0/1078431018 conn(0x7fb5c8077d30 msgr2=0x7fb5c810d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:32.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.090+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 shutdown_connections 2024-04-26T20:11:32.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:31.091+0000 7fb5c7e00640 1 -- 172.21.15.94:0/1078431018 wait complete. 2024-04-26T20:11:32.080 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:11:32.093 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:31 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1078431018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:11:32.100 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:30 smithi204 ceph-mon[32618]: pgmap v425: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:32.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:30 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/945756535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:11:32.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:31 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1078431018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:11:32.169 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:31 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1078431018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:11:32.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.308+0000 7f7b57e00640 1 -- 172.21.15.94:0/2811556793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b58103ab0 msgr2=0x7f7b58103f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:32.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.308+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2811556793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b58103ab0 0x7f7b58103f30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009900 tx=0x7f7b4002a410 comp rx=0 tx=0).stop 2024-04-26T20:11:32.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.308+0000 7f7b57e00640 1 -- 172.21.15.94:0/2811556793 shutdown_connections 2024-04-26T20:11:32.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.308+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2811556793 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b58104470 0x7f7b5810ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:32.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.308+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2811556793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b58103ab0 0x7f7b58103f30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:32.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.308+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2811556793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b581028c0 0x7f7b58102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:32.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.308+0000 7f7b57e00640 1 -- 172.21.15.94:0/2811556793 >> 172.21.15.94:0/2811556793 conn(0x7f7b580fde30 msgr2=0x7f7b58100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.308+0000 7f7b57e00640 1 -- 172.21.15.94:0/2811556793 shutdown_connections 2024-04-26T20:11:32.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.308+0000 7f7b57e00640 1 -- 172.21.15.94:0/2811556793 wait complete. 2024-04-26T20:11:32.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.309+0000 7f7b57e00640 1 Processor -- start 2024-04-26T20:11:32.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.309+0000 7f7b57e00640 1 -- start start 2024-04-26T20:11:32.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b57e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b581028c0 0x7f7b58198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b57e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b58103ab0 0x7f7b58199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:32.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b58104470 0x7f7b581a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:32.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b57e00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5810e380 con 0x7f7b581028c0 2024-04-26T20:11:32.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b57e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5810e200 con 0x7f7b58103ab0 2024-04-26T20:11:32.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b57e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5810e500 con 0x7f7b58104470 2024-04-26T20:11:32.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b56a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b581028c0 0x7f7b58198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:32.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b56a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b581028c0 0x7f7b58198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57926/0 (socket says 172.21.15.94:57926) 2024-04-26T20:11:32.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b56a00640 1 -- 172.21.15.94:0/2528841042 learned_addr learned my addr 172.21.15.94:0/2528841042 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:32.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b4e000640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b58103ab0 0x7f7b58199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:32.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b57400640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b58104470 0x7f7b581a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:32.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b56a00640 1 -- 172.21.15.94:0/2528841042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b58104470 msgr2=0x7f7b581a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:32.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b56a00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b58104470 0x7f7b581a0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:32.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b56a00640 1 -- 172.21.15.94:0/2528841042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b58103ab0 msgr2=0x7f7b58199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:32.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b56a00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b58103ab0 0x7f7b58199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:32.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.310+0000 7f7b56a00640 1 -- 172.21.15.94:0/2528841042 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b581a0700 con 0x7f7b581028c0 2024-04-26T20:11:32.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.311+0000 7f7b56a00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b581028c0 0x7f7b58198ae0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800e9c0 tx=0x7f7b4800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:32.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.311+0000 7f7b4fe00640 1 -- 172.21.15.94:0/2528841042 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4800cd80 con 0x7f7b581028c0 2024-04-26T20:11:32.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.311+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b581a09f0 con 0x7f7b581028c0 2024-04-26T20:11:32.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.311+0000 7f7b4fe00640 1 -- 172.21.15.94:0/2528841042 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b480035f0 con 0x7f7b581028c0 2024-04-26T20:11:32.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.311+0000 7f7b4fe00640 1 -- 172.21.15.94:0/2528841042 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48010690 con 0x7f7b581028c0 2024-04-26T20:11:32.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.311+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b581a0f10 con 0x7f7b581028c0 2024-04-26T20:11:32.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.313+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b58069670 con 0x7f7b581028c0 2024-04-26T20:11:32.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.315+0000 7f7b4fe00640 1 -- 172.21.15.94:0/2528841042 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7b48004000 con 0x7f7b581028c0 2024-04-26T20:11:32.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.316+0000 7f7b4fe00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7b3c0a3c40 0x7f7b3c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:32.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.316+0000 7f7b4fe00640 1 -- 172.21.15.94:0/2528841042 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f7b480d0eb0 con 0x7f7b581028c0 2024-04-26T20:11:33.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.316+0000 7f7b4e000640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7b3c0a3c40 0x7f7b3c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:33.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.317+0000 7f7b4e000640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7b3c0a3c40 0x7f7b3c0a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7b40000c00 tx=0x7f7b400314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:33.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.322+0000 7f7b4fe00640 1 -- 172.21.15.94:0/2528841042 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7b4809d300 con 0x7f7b581028c0 2024-04-26T20:11:33.044 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:32 smithi103 ceph-mon[32635]: pgmap v426: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:33.049 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:32 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2528841042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:11:33.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.523+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7b5802dbe0 con 0x7f7b581028c0 2024-04-26T20:11:33.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.524+0000 7f7b4fe00640 1 -- 172.21.15.94:0/2528841042 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f7b4809cc00 con 0x7f7b581028c0 2024-04-26T20:11:33.066 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:11:33.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.526+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7b3c0a3c40 msgr2=0x7f7b3c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:33.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.526+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7b3c0a3c40 0x7f7b3c0a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7b40000c00 tx=0x7f7b400314a0 comp rx=0 tx=0).stop 2024-04-26T20:11:33.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.526+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b581028c0 msgr2=0x7f7b58198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:33.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.526+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b581028c0 0x7f7b58198ae0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800e9c0 tx=0x7f7b4800ee80 comp rx=0 tx=0).stop 2024-04-26T20:11:33.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.527+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 shutdown_connections 2024-04-26T20:11:33.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.527+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7b3c0a3c40 0x7f7b3c0a6100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.527+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b58104470 0x7f7b581a0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:33.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.527+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b58103ab0 0x7f7b58199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.527+0000 7f7b57e00640 1 --2- 172.21.15.94:0/2528841042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b581028c0 0x7f7b58198ae0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:33.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.527+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 >> 172.21.15.94:0/2528841042 conn(0x7f7b580fde30 msgr2=0x7f7b58106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:33.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.527+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 shutdown_connections 2024-04-26T20:11:33.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:32.527+0000 7f7b57e00640 1 -- 172.21.15.94:0/2528841042 wait complete. 2024-04-26T20:11:33.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:11:33.367 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:32 smithi094 ceph-mon[28050]: pgmap v426: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:33.396 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:32 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2528841042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:11:33.406 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:32 smithi204 ceph-mon[32618]: pgmap v426: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:33.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:32 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2528841042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:11:33.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.616+0000 7f4dcc400640 1 -- 172.21.15.94:0/3491301081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 msgr2=0x7f4dc41093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:33.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.616+0000 7f4dcc400640 1 --2- 172.21.15.94:0/3491301081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 0x7f4dc41093b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009900 tx=0x7f4db802a400 comp rx=0 tx=0).stop 2024-04-26T20:11:33.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.616+0000 7f4dcc400640 1 -- 172.21.15.94:0/3491301081 shutdown_connections 2024-04-26T20:11:33.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.616+0000 7f4dcc400640 1 --2- 172.21.15.94:0/3491301081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc410ab40 0x7f4dc41113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:33.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.616+0000 7f4dcc400640 1 --2- 172.21.15.94:0/3491301081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dc410a180 0x7f4dc410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:33.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.616+0000 7f4dcc400640 1 --2- 172.21.15.94:0/3491301081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 0x7f4dc41093b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:33.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.616+0000 7f4dcc400640 1 -- 172.21.15.94:0/3491301081 >> 172.21.15.94:0/3491301081 conn(0x7f4dc4077f40 msgr2=0x7f4dc407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:33.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.616+0000 7f4dcc400640 1 -- 172.21.15.94:0/3491301081 shutdown_connections 2024-04-26T20:11:33.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.616+0000 7f4dcc400640 1 -- 172.21.15.94:0/3491301081 wait complete. 2024-04-26T20:11:33.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.617+0000 7f4dcc400640 1 Processor -- start 2024-04-26T20:11:33.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dcc400640 1 -- start start 2024-04-26T20:11:33.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dcc400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 0x7f4dc419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:33.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dcc400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dc410a180 0x7f4dc419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:33.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dcc400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc410ab40 0x7f4dc41a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:33.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dcc400640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc4114b20 con 0x7f4dc4108f90 2024-04-26T20:11:33.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dcc400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc41149a0 con 0x7f4dc410ab40 2024-04-26T20:11:33.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dcc400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc4114ca0 con 0x7f4dc410a180 2024-04-26T20:11:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dc9a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 0x7f4dc419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:34.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dc9a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 0x7f4dc419f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57942/0 (socket says 172.21.15.94:57942) 2024-04-26T20:11:34.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dc9a00640 1 -- 172.21.15.94:0/1407119059 learned_addr learned my addr 172.21.15.94:0/1407119059 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:34.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dca400640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc410ab40 0x7f4dc41a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:34.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.618+0000 7f4dc9000640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dc410a180 0x7f4dc419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dc9a00640 1 -- 172.21.15.94:0/1407119059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dc410a180 msgr2=0x7f4dc419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dc9a00640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dc410a180 0x7f4dc419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:34.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dc9a00640 1 -- 172.21.15.94:0/1407119059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc410ab40 msgr2=0x7f4dc41a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:34.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dc9a00640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc410ab40 0x7f4dc41a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dc9a00640 1 -- 172.21.15.94:0/1407119059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc41a6da0 con 0x7f4dc4108f90 2024-04-26T20:11:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dc9a00640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 0x7f4dc419f180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4db8031500 tx=0x7f4db8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dc2000640 1 -- 172.21.15.94:0/1407119059 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db803f070 con 0x7f4dc4108f90 2024-04-26T20:11:34.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dc2000640 1 -- 172.21.15.94:0/1407119059 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4db8031c50 con 0x7f4dc4108f90 2024-04-26T20:11:34.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dc2000640 1 -- 172.21.15.94:0/1407119059 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db8035030 con 0x7f4dc4108f90 2024-04-26T20:11:34.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc41a7030 con 0x7f4dc4108f90 2024-04-26T20:11:34.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.619+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc41a7520 con 0x7f4dc4108f90 2024-04-26T20:11:34.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.620+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dc4110330 con 0x7f4dc4108f90 2024-04-26T20:11:34.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.622+0000 7f4dc2000640 1 -- 172.21.15.94:0/1407119059 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4db8042070 con 0x7f4dc4108f90 2024-04-26T20:11:34.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.624+0000 7f4dc2000640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4dac0a3c40 0x7f4dac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:34.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.624+0000 7f4dc2000640 1 -- 172.21.15.94:0/1407119059 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f4db80f17e0 con 0x7f4dc4108f90 2024-04-26T20:11:34.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.624+0000 7f4dc9000640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4dac0a3c40 0x7f4dac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:34.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.625+0000 7f4dc9000640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4dac0a3c40 0x7f4dac0a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4dc41a0690 tx=0x7f4db0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:34.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.626+0000 7f4dc2000640 1 -- 172.21.15.94:0/1407119059 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4db80bd5e0 con 0x7f4dc4108f90 2024-04-26T20:11:34.409 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:33 smithi204 ceph-mon[32618]: pgmap v427: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:34.451 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:33 smithi103 ceph-mon[32635]: pgmap v427: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:34.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.868+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4dc402dad0 con 0x7f4dc4108f90 2024-04-26T20:11:34.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.870+0000 7f4dc2000640 1 -- 172.21.15.94:0/1407119059 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11878 (secure 0 0 0) 0x7f4db80c18e0 con 0x7f4dc4108f90 2024-04-26T20:11:34.514 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:11:34.520 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:09:27.103941+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24562,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24562,24541]},"id":1}]} 2024-04-26T20:11:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-26T20:11:34.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4dac0a3c40 msgr2=0x7f4dac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4dac0a3c40 0x7f4dac0a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4dc41a0690 tx=0x7f4db0005f90 comp rx=0 tx=0).stop 2024-04-26T20:11:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 msgr2=0x7f4dc419f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 0x7f4dc419f180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4db8031500 tx=0x7f4db8003c10 comp rx=0 tx=0).stop 2024-04-26T20:11:34.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 shutdown_connections 2024-04-26T20:11:34.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4dac0a3c40 0x7f4dac0a6100 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dc410ab40 0x7f4dc41a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:34.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dc410a180 0x7f4dc419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:34.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 --2- 172.21.15.94:0/1407119059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4dc4108f90 0x7f4dc419f180 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:34.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 >> 172.21.15.94:0/1407119059 conn(0x7f4dc4077f40 msgr2=0x7f4dc410d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:34.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 shutdown_connections 2024-04-26T20:11:34.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:33.873+0000 7f4dcc400640 1 -- 172.21.15.94:0/1407119059 wait complete. 2024-04-26T20:11:34.777 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:11:34.783 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:33 smithi094 ceph-mon[28050]: pgmap v427: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:34.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.994+0000 7f68e4800640 1 -- 172.21.15.94:0/1689881225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10a180 msgr2=0x7f68dc10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:35.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.994+0000 7f68e4800640 1 --2- 172.21.15.94:0/1689881225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10a180 0x7f68dc10a600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009900 tx=0x7f68c802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:11:35.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.995+0000 7f68e4800640 1 -- 172.21.15.94:0/1689881225 shutdown_connections 2024-04-26T20:11:35.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.995+0000 7f68e4800640 1 --2- 172.21.15.94:0/1689881225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68dc10ab40 0x7f68dc1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:35.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.995+0000 7f68e4800640 1 --2- 172.21.15.94:0/1689881225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10a180 0x7f68dc10a600 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:35.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.995+0000 7f68e4800640 1 --2- 172.21.15.94:0/1689881225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68dc108f90 0x7f68dc1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:35.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.995+0000 7f68e4800640 1 -- 172.21.15.94:0/1689881225 >> 172.21.15.94:0/1689881225 conn(0x7f68dc077f40 msgr2=0x7f68dc07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:35.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.995+0000 7f68e4800640 1 -- 172.21.15.94:0/1689881225 shutdown_connections 2024-04-26T20:11:35.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.995+0000 7f68e4800640 1 -- 172.21.15.94:0/1689881225 wait complete. 2024-04-26T20:11:35.223 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:34 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1407119059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:11:35.232 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:34 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1407119059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:11:35.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e4800640 1 Processor -- start 2024-04-26T20:11:35.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e4800640 1 -- start start 2024-04-26T20:11:35.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e4800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68dc108f90 0x7f68dc19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:35.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e4800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68dc10a180 0x7f68dc19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:35.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e4800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10ab40 0x7f68dc1a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:35.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e4800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc114ad0 con 0x7f68dc10ab40 2024-04-26T20:11:35.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e4800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc114950 con 0x7f68dc10a180 2024-04-26T20:11:35.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e4800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc114c50 con 0x7f68dc108f90 2024-04-26T20:11:35.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e2800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10ab40 0x7f68dc1a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:35.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e1e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68dc108f90 0x7f68dc19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:35.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e2800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10ab40 0x7f68dc1a6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57952/0 (socket says 172.21.15.94:57952) 2024-04-26T20:11:35.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e2800640 1 -- 172.21.15.94:0/2048335427 learned_addr learned my addr 172.21.15.94:0/2048335427 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:35.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.996+0000 7f68e1400640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68dc10a180 0x7f68dc19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:35.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68e1e00640 1 -- 172.21.15.94:0/2048335427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68dc10a180 msgr2=0x7f68dc19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:35.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68e1e00640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68dc10a180 0x7f68dc19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:35.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68e1e00640 1 -- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10ab40 msgr2=0x7f68dc1a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:35.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68e1e00640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10ab40 0x7f68dc1a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:35.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68e1e00640 1 -- 172.21.15.94:0/2048335427 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68dc1a6d70 con 0x7f68dc108f90 2024-04-26T20:11:35.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68e2800640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10ab40 0x7f68dc1a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:35.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68e1e00640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68dc108f90 0x7f68dc19f150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f68d000e9c0 tx=0x7f68d000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:35.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68d6a00640 1 -- 172.21.15.94:0/2048335427 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d000cd80 con 0x7f68dc108f90 2024-04-26T20:11:35.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68d6a00640 1 -- 172.21.15.94:0/2048335427 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68d00108e0 con 0x7f68dc108f90 2024-04-26T20:11:35.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68dc1a7060 con 0x7f68dc108f90 2024-04-26T20:11:35.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.997+0000 7f68d6a00640 1 -- 172.21.15.94:0/2048335427 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d00187f0 con 0x7f68dc108f90 2024-04-26T20:11:35.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.998+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68dc1a7580 con 0x7f68dc108f90 2024-04-26T20:11:35.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:34.999+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68dc1102b0 con 0x7f68dc108f90 2024-04-26T20:11:35.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.001+0000 7f68d6a00640 1 -- 172.21.15.94:0/2048335427 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f68d001f020 con 0x7f68dc108f90 2024-04-26T20:11:35.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.002+0000 7f68d6a00640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68c40a3c40 0x7f68c40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:35.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.002+0000 7f68d6a00640 1 -- 172.21.15.94:0/2048335427 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f68d00d0b30 con 0x7f68dc108f90 2024-04-26T20:11:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.002+0000 7f68e1400640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68c40a3c40 0x7f68c40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:35.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.003+0000 7f68e1400640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68c40a3c40 0x7f68c40a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f68c8000c00 tx=0x7f68c80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:35.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.004+0000 7f68d6a00640 1 -- 172.21.15.94:0/2048335427 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f68d009cf80 con 0x7f68dc108f90 2024-04-26T20:11:35.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.204+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68dc063530 con 0x7f68dc108f90 2024-04-26T20:11:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.206+0000 7f68d6a00640 1 -- 172.21.15.94:0/2048335427 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24879 (secure 0 0 0) 0x7f68d009c880 con 0x7f68dc108f90 2024-04-26T20:11:35.964 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:11:35.976 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:10:13.755106+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:11:35.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68c40a3c40 msgr2=0x7f68c40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68c40a3c40 0x7f68c40a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f68c8000c00 tx=0x7f68c80314a0 comp rx=0 tx=0).stop 2024-04-26T20:11:36.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68dc108f90 msgr2=0x7f68dc19f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:36.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68dc108f90 0x7f68dc19f150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f68d000e9c0 tx=0x7f68d000ee80 comp rx=0 tx=0).stop 2024-04-26T20:11:36.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 shutdown_connections 2024-04-26T20:11:36.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f68c40a3c40 0x7f68c40a6100 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:36.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68dc10ab40 0x7f68dc1a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:36.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f68dc10a180 0x7f68dc19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:36.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 --2- 172.21.15.94:0/2048335427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68dc108f90 0x7f68dc19f150 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:36.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.209+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 >> 172.21.15.94:0/2048335427 conn(0x7f68dc077f40 msgr2=0x7f68dc10d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.210+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 shutdown_connections 2024-04-26T20:11:36.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:35.210+0000 7f68e4800640 1 -- 172.21.15.94:0/2048335427 wait complete. 2024-04-26T20:11:36.234 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:34 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1407119059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:11:36.246 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:11:36.256 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:35 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2048335427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:11:36.264 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:35 smithi094 ceph-mon[28050]: pgmap v428: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:36.276 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:35 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2048335427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:11:36.286 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:35 smithi204 ceph-mon[32618]: pgmap v428: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:36.294 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:35 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2048335427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:11:36.354 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:35 smithi103 ceph-mon[32635]: pgmap v428: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.486+0000 7f022ee00640 1 -- 172.21.15.94:0/2155496599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f022810a940 msgr2=0x7f02281111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:36.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.486+0000 7f022ee00640 1 --2- 172.21.15.94:0/2155496599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f022810a940 0x7f02281111d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0220009ca0 tx=0x7f022002a8c0 comp rx=0 tx=0).stop 2024-04-26T20:11:36.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.486+0000 7f022ee00640 1 -- 172.21.15.94:0/2155496599 shutdown_connections 2024-04-26T20:11:36.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.486+0000 7f022ee00640 1 --2- 172.21.15.94:0/2155496599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f022810a940 0x7f02281111d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:36.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.486+0000 7f022ee00640 1 --2- 172.21.15.94:0/2155496599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0228109f80 0x7f022810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:36.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.486+0000 7f022ee00640 1 --2- 172.21.15.94:0/2155496599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0228108d90 0x7f02281091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.486+0000 7f022ee00640 1 -- 172.21.15.94:0/2155496599 >> 172.21.15.94:0/2155496599 conn(0x7f0228077d30 msgr2=0x7f022807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.486+0000 7f022ee00640 1 -- 172.21.15.94:0/2155496599 shutdown_connections 2024-04-26T20:11:36.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.486+0000 7f022ee00640 1 -- 172.21.15.94:0/2155496599 wait complete. 2024-04-26T20:11:36.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.487+0000 7f022ee00640 1 Processor -- start 2024-04-26T20:11:36.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022ee00640 1 -- start start 2024-04-26T20:11:36.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0228108d90 0x7f022819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:36.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022ee00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0228109f80 0x7f022819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:36.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022ee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022810a940 0x7f02281a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:36.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022ee00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0228114910 con 0x7f0228109f80 2024-04-26T20:11:36.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0228114790 con 0x7f0228108d90 2024-04-26T20:11:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022ee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0228114a90 con 0x7f022810a940 2024-04-26T20:11:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022d000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0228109f80 0x7f022819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:36.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022d000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0228109f80 0x7f022819f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57976/0 (socket says 172.21.15.94:57976) 2024-04-26T20:11:36.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022d000640 1 -- 172.21.15.94:0/1383111344 learned_addr learned my addr 172.21.15.94:0/1383111344 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:36.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022da00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0228108d90 0x7f022819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:36.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.488+0000 7f022e400640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022810a940 0x7f02281a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:36.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f022d000640 1 -- 172.21.15.94:0/1383111344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022810a940 msgr2=0x7f02281a64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:37.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f022d000640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022810a940 0x7f02281a64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:37.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f022d000640 1 -- 172.21.15.94:0/1383111344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0228108d90 msgr2=0x7f022819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:37.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f022d000640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0228108d90 0x7f022819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:37.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f022d000640 1 -- 172.21.15.94:0/1383111344 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02281a6b40 con 0x7f0228109f80 2024-04-26T20:11:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f022da00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0228108d90 0x7f022819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:37.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f022d000640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0228109f80 0x7f022819f460 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f021400e9f0 tx=0x7f021400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:37.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f0226000640 1 -- 172.21.15.94:0/1383111344 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021400cd80 con 0x7f0228109f80 2024-04-26T20:11:37.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f0226000640 1 -- 172.21.15.94:0/1383111344 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02140108e0 con 0x7f0228109f80 2024-04-26T20:11:37.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02281a6e30 con 0x7f0228109f80 2024-04-26T20:11:37.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f0226000640 1 -- 172.21.15.94:0/1383111344 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0214018980 con 0x7f0228109f80 2024-04-26T20:11:37.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.489+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02281a72f0 con 0x7f0228109f80 2024-04-26T20:11:37.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.490+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02281100b0 con 0x7f0228109f80 2024-04-26T20:11:37.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.492+0000 7f0226000640 1 -- 172.21.15.94:0/1383111344 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0214010430 con 0x7f0228109f80 2024-04-26T20:11:37.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.495+0000 7f0226000640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02100a3c40 0x7f02100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:37.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.495+0000 7f0226000640 1 -- 172.21.15.94:0/1383111344 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f02140d0d20 con 0x7f0228109f80 2024-04-26T20:11:37.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.495+0000 7f022da00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02100a3c40 0x7f02100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:37.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.496+0000 7f022da00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02100a3c40 0x7f02100a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f021c005fd0 tx=0x7f021c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:37.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.499+0000 7f0226000640 1 -- 172.21.15.94:0/1383111344 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f021409d170 con 0x7f0228109f80 2024-04-26T20:11:37.395 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:36 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1383111344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:11:37.451 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:36 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1383111344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:11:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.746+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f022802dcd0 con 0x7f0228109f80 2024-04-26T20:11:37.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.748+0000 7f0226000640 1 -- 172.21.15.94:0/1383111344 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11878 (secure 0 0 0) 0x7f021409ca70 con 0x7f0228109f80 2024-04-26T20:11:37.476 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:11:37.485 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:09:27.103941+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24562,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24562,24541]},"id":1}]} 2024-04-26T20:11:37.496 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-26T20:11:37.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02100a3c40 msgr2=0x7f02100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02100a3c40 0x7f02100a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f021c005fd0 tx=0x7f021c00b390 comp rx=0 tx=0).stop 2024-04-26T20:11:37.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0228109f80 msgr2=0x7f022819f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:37.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0228109f80 0x7f022819f460 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f021400e9f0 tx=0x7f021400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:11:37.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 shutdown_connections 2024-04-26T20:11:37.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02100a3c40 0x7f02100a6100 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:37.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f022810a940 0x7f02281a64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0228109f80 0x7f022819f460 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:37.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 --2- 172.21.15.94:0/1383111344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0228108d90 0x7f022819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 >> 172.21.15.94:0/1383111344 conn(0x7f0228077d30 msgr2=0x7f022810d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:37.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 shutdown_connections 2024-04-26T20:11:37.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:36.751+0000 7f022ee00640 1 -- 172.21.15.94:0/1383111344 wait complete. 2024-04-26T20:11:37.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T20:08:18.399846+0000', 'modified': '2024-04-26T20:09:27.103941+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24575, 'mds_1': 24562, 'mds_2': 24541}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6835/4163398350', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6834', 'nonce': 4163398350}, {'type': 'v1', 'addr': '172.21.15.204:6835', 'nonce': 4163398350}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6837/820565196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6836', 'nonce': 820565196}, {'type': 'v1', 'addr': '172.21.15.204:6837', 'nonce': 820565196}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6837/3697356689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 3697356689}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 3697356689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.204:6839/3633033106', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6838', 'nonce': 3633033106}, {'type': 'v1', 'addr': '172.21.15.204:6839', 'nonce': 3633033106}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24562': {'gid': 24562, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.94:6841/989331046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6840', 'nonce': 989331046}, {'type': 'v1', 'addr': '172.21.15.94:6841', 'nonce': 989331046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.103:6839/1048742250', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 1048742250}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 1048742250}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24575, 'qdb_cluster': [24575, 24562, 24541]} 2024-04-26T20:11:37.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T20:11:37.773 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-26T20:11:37.824 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:36 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1383111344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:11:38.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.066+0000 7f1fb2000640 1 -- 172.21.15.94:0/3049838555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fac106740 msgr2=0x7f1fac10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:38.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.066+0000 7f1fb2000640 1 --2- 172.21.15.94:0/3049838555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fac106740 0x7f1fac10cfd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009ca0 tx=0x7f1fa402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:11:38.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.066+0000 7f1fb2000640 1 -- 172.21.15.94:0/3049838555 shutdown_connections 2024-04-26T20:11:38.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.066+0000 7f1fb2000640 1 --2- 172.21.15.94:0/3049838555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fac106740 0x7f1fac10cfd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.066+0000 7f1fb2000640 1 --2- 172.21.15.94:0/3049838555 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fac105d80 0x7f1fac106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:38.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.066+0000 7f1fb2000640 1 --2- 172.21.15.94:0/3049838555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fac104b90 0x7f1fac104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:38.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.066+0000 7f1fb2000640 1 -- 172.21.15.94:0/3049838555 >> 172.21.15.94:0/3049838555 conn(0x7f1fac100120 msgr2=0x7f1fac102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:38.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.067+0000 7f1fb2000640 1 -- 172.21.15.94:0/3049838555 shutdown_connections 2024-04-26T20:11:38.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.067+0000 7f1fb2000640 1 -- 172.21.15.94:0/3049838555 wait complete. 2024-04-26T20:11:38.221 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:37 smithi204 ceph-mon[32618]: pgmap v429: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:38.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fb2000640 1 Processor -- start 2024-04-26T20:11:38.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fb2000640 1 -- start start 2024-04-26T20:11:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fb2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fac104b90 0x7f1fac19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:38.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fb2000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fac105d80 0x7f1fac19b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:38.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fb2000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fac106740 0x7f1fac1a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:38.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fb2000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac1106f0 con 0x7f1fac105d80 2024-04-26T20:11:38.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fb2000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac110570 con 0x7f1fac106740 2024-04-26T20:11:38.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fb2000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fac110870 con 0x7f1fac104b90 2024-04-26T20:11:38.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fa3e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fac105d80 0x7f1fac19b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:38.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fa3e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fac105d80 0x7f1fac19b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58002/0 (socket says 172.21.15.94:58002) 2024-04-26T20:11:38.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fa3e00640 1 -- 172.21.15.94:0/1512628029 learned_addr learned my addr 172.21.15.94:0/1512628029 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:38.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.068+0000 7f1fabe00640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fac106740 0x7f1fac1a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:38.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.069+0000 7f1fab400640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fac104b90 0x7f1fac19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:38.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.069+0000 7f1fa3e00640 1 -- 172.21.15.94:0/1512628029 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fac104b90 msgr2=0x7f1fac19ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:38.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.069+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fac104b90 0x7f1fac19ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:38.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.069+0000 7f1fa3e00640 1 -- 172.21.15.94:0/1512628029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fac106740 msgr2=0x7f1fac1a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:38.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.069+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fac106740 0x7f1fac1a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:38.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.069+0000 7f1fa3e00640 1 -- 172.21.15.94:0/1512628029 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fac1a2940 con 0x7f1fac105d80 2024-04-26T20:11:38.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.069+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fac105d80 0x7f1fac19b270 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e9f0 tx=0x7f1f9400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:38.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.069+0000 7f1fa8c00640 1 -- 172.21.15.94:0/1512628029 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9400cd80 con 0x7f1fac105d80 2024-04-26T20:11:38.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.069+0000 7f1fb2000640 1 -- 172.21.15.94:0/1512628029 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fac1a2c30 con 0x7f1fac105d80 2024-04-26T20:11:38.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.070+0000 7f1fa8c00640 1 -- 172.21.15.94:0/1512628029 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f940108e0 con 0x7f1fac105d80 2024-04-26T20:11:38.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.070+0000 7f1fa8c00640 1 -- 172.21.15.94:0/1512628029 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f94018980 con 0x7f1fac105d80 2024-04-26T20:11:38.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.070+0000 7f1fb2000640 1 -- 172.21.15.94:0/1512628029 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fac1a31d0 con 0x7f1fac105d80 2024-04-26T20:11:38.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.073+0000 7f1fb4e07740 1 Processor -- start 2024-04-26T20:11:38.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.073+0000 7f1fa8c00640 1 -- 172.21.15.94:0/1512628029 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1f94010430 con 0x7f1fac105d80 2024-04-26T20:11:38.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.073+0000 7f1fb4e07740 1 -- start start 2024-04-26T20:11:38.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.073+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84ea510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:38.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.073+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 0x564fb84ef5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:38.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.073+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb84f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:38.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.073+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564fb844e600 con 0x564fb84ea0f0 2024-04-26T20:11:38.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.073+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564fb83eb100 con 0x564fb84efc50 2024-04-26T20:11:38.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.073+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564fb844a5c0 con 0x564fb84eaae0 2024-04-26T20:11:38.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fabe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84ea510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fa3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb84f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:38.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fab400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 0x564fb84ef5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:38.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fa3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb84f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:48172/0 (socket says 172.21.15.94:48172) 2024-04-26T20:11:38.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fa3e00640 1 -- 172.21.15.94:0/2987393696 learned_addr learned my addr 172.21.15.94:0/2987393696 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:38.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fa3e00640 1 -- 172.21.15.94:0/2987393696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 msgr2=0x564fb84ef5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:38.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/2987393696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 0x564fb84ef5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:38.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fa3e00640 1 -- 172.21.15.94:0/2987393696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 msgr2=0x564fb84ea510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:38.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/2987393696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84ea510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:38.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.074+0000 7f1fa3e00640 1 -- 172.21.15.94:0/2987393696 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fb81602d0 con 0x564fb84efc50 2024-04-26T20:11:39.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.075+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/2987393696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb84f20e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1f940d4e90 tx=0x7f1f940d97e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5699c4d1b85f03e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.075+0000 7f1fa8c00640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1f900a3c40 0x7f1f900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:39.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.075+0000 7f1f9be00640 1 -- 172.21.15.94:0/2987393696 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f94098d30 con 0x564fb84efc50 2024-04-26T20:11:39.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.075+0000 7f1f9be00640 1 -- 172.21.15.94:0/2987393696 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f94010720 con 0x564fb84efc50 2024-04-26T20:11:39.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.075+0000 7f1f9be00640 1 -- 172.21.15.94:0/2987393696 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9409c2a0 con 0x564fb84efc50 2024-04-26T20:11:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.075+0000 7f1fa8c00640 1 -- 172.21.15.94:0/1512628029 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f1f940d0d20 con 0x7f1fac105d80 2024-04-26T20:11:39.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.075+0000 7f1fabe00640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1f900a3c40 0x7f1f900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:39.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 -- 172.21.15.94:0/2987393696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 msgr2=0x564fb84f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:39.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/2987393696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb84f20e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1f940d4e90 tx=0x7f1f940d97e0 comp rx=0 tx=0).stop 2024-04-26T20:11:39.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fabe00640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1f900a3c40 0x7f1f900a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4005e30 tx=0x7f1fa4031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 -- 172.21.15.94:0/2987393696 shutdown_connections 2024-04-26T20:11:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/2987393696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb84f20e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:39.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/2987393696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 0x564fb84ef5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/2987393696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84ea510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:39.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 -- 172.21.15.94:0/2987393696 >> 172.21.15.94:0/2987393696 conn(0x564fb847ed10 msgr2=0x564fb84e5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:39.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 -- 172.21.15.94:0/2987393696 shutdown_connections 2024-04-26T20:11:39.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 -- 172.21.15.94:0/2987393696 wait complete. 2024-04-26T20:11:39.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.076+0000 7f1fb4e07740 1 -- start start 2024-04-26T20:11:39.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fb4e07740 1 Processor -- start 2024-04-26T20:11:39.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:39.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:39.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84fba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:39.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564fb84f2c00 con 0x564fb84ea0f0 2024-04-26T20:11:39.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564fb83d5f70 con 0x564fb84eaae0 2024-04-26T20:11:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564fb84f2d80 con 0x564fb84efc50 2024-04-26T20:11:39.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fb4e07740 10 client.0 _ll_register_callbacks cb 0x564fb8444480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:11:39.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fab400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fa3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:39.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fa3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:48174/0 (socket says 172.21.15.94:48174) 2024-04-26T20:11:39.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.077+0000 7f1fa3e00640 1 -- 172.21.15.94:0/1360297595 learned_addr learned my addr 172.21.15.94:0/1360297595 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:39.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1fabe00640 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84fba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:39.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1fa3e00640 1 -- 172.21.15.94:0/1360297595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 msgr2=0x564fb84fba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:39.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84fba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:39.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1fa3e00640 1 -- 172.21.15.94:0/1360297595 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 msgr2=0x564fb84f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:39.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:39.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1fa3e00640 1 -- 172.21.15.94:0/1360297595 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fb822a6b0 con 0x564fb84eaae0 2024-04-26T20:11:39.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84f4a20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1f9409cab0 tx=0x7f1f94018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6c5a09c76534b68 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:39.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1f98c00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-26T20:11:39.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1f98c00640 1 -- 172.21.15.94:0/1360297595 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f940d3020 con 0x564fb84eaae0 2024-04-26T20:11:39.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1f98c00640 1 -- 172.21.15.94:0/1360297595 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f1f94010a90 con 0x564fb84eaae0 2024-04-26T20:11:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.078+0000 7f1fb4e07740 1 -- 172.21.15.94:0/1360297595 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564fb845c210 con 0x564fb84eaae0 2024-04-26T20:11:39.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1f98c00640 1 -- 172.21.15.94:0/1360297595 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1f9409c480 con 0x564fb84eaae0 2024-04-26T20:11:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1fb4e07740 10 client.25151 fetch_fsmap learned FSMap version 24 2024-04-26T20:11:39.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1fb4e07740 1 -- 172.21.15.94:0/1360297595 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564fb84f2c00 con 0x564fb84eaae0 2024-04-26T20:11:39.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1f98c00640 1 -- 172.21.15.94:0/1360297595 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564fb845c210 con 0x564fb84eaae0 2024-04-26T20:11:39.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1f98c00640 1 -- 172.21.15.94:0/1360297595 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1f940e0070 con 0x564fb84eaae0 2024-04-26T20:11:39.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1fb4e07740 10 client.25151 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:11:39.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1fb4e07740 10 client.25151 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T20:11:39.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1fb4e07740 20 client.25151 populate_metadata read hostname 'smithi094' 2024-04-26T20:11:39.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x564fb8500b80 0x564fb8501ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:39.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1fb4e07740 4 client.25151 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:11:39.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.079+0000 7f1fb4e07740 1 -- 172.21.15.94:0/1360297595 --> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fb8456080 con 0x564fb8500b80 2024-04-26T20:11:39.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.080+0000 7f1fab400640 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x564fb8500b80 0x564fb8501ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:39.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.080+0000 7f1fab400640 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x564fb8500b80 0x564fb8501ff0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=afa7152ac623e0cc server_cookie=87e879abbf784506 in_seq=0 out_seq=0 2024-04-26T20:11:39.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.080+0000 7f1f98c00640 10 client.25151 ms_handle_connect on v2:172.21.15.204:6838/3633033106 2024-04-26T20:11:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1f98c00640 1 -- 172.21.15.94:0/1360297595 <== mds.2 v2:172.21.15.204:6838/3633033106 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564fb8456080 con 0x564fb8500b80 2024-04-26T20:11:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1f98c00640 10 client.25151 handle_command_reply: tid=0 2024-04-26T20:11:39.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1fb4e07740 1 client.25151 shutdown 2024-04-26T20:11:39.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1fb4e07740 1 -- 172.21.15.94:0/1360297595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 msgr2=0x564fb84f4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:39.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84f4a20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1f9409cab0 tx=0x7f1f94018420 comp rx=0 tx=0).stop 2024-04-26T20:11:39.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1fb4e07740 1 -- 172.21.15.94:0/1360297595 shutdown_connections 2024-04-26T20:11:40.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x564fb8500b80 0x564fb8501ff0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84fba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1f98c00640 0 client.25151 ms_handle_reset on v2:172.21.15.204:6838/3633033106 2024-04-26T20:11:40.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84f4a20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/1360297595 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.081+0000 7f1fb4e07740 1 -- 172.21.15.94:0/1360297595 >> 172.21.15.94:0/1360297595 conn(0x564fb847ed10 msgr2=0x564fb84e4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:40.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.082+0000 7f1fb4e07740 1 -- 172.21.15.94:0/1360297595 shutdown_connections 2024-04-26T20:11:40.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.082+0000 7f1fb4e07740 1 -- 172.21.15.94:0/1360297595 wait complete. 2024-04-26T20:11:40.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.082+0000 7f1fb4e07740 20 client.25151 trim_cache size 0 max 16384 2024-04-26T20:11:40.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.101+0000 7f1fb4e07740 1 Processor -- start 2024-04-26T20:11:40.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fb4e07740 1 -- start start 2024-04-26T20:11:40.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb85092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:40.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 0x564fb8509800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:40.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb8509d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:40.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564fb8500330 con 0x564fb84ea0f0 2024-04-26T20:11:40.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564fb8500a00 con 0x564fb84efc50 2024-04-26T20:11:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564fb85010d0 con 0x564fb84eaae0 2024-04-26T20:11:40.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fab400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb85092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:40.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fabe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb8509d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:40.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fab400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb85092c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58032/0 (socket says 172.21.15.94:58032) 2024-04-26T20:11:40.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fab400640 1 -- 172.21.15.94:0/3711355612 learned_addr learned my addr 172.21.15.94:0/3711355612 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:40.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.102+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/3711355612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 0x564fb8509800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.103+0000 7f1fab400640 1 -- 172.21.15.94:0/3711355612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 msgr2=0x564fb8509800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.103+0000 7f1fab400640 1 --2- 172.21.15.94:0/3711355612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 0x564fb8509800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.103+0000 7f1fab400640 1 -- 172.21.15.94:0/3711355612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 msgr2=0x564fb8509d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:40.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.103+0000 7f1fab400640 1 --2- 172.21.15.94:0/3711355612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb8509d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.103+0000 7f1fab400640 1 -- 172.21.15.94:0/3711355612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fb84ef930 con 0x564fb84ea0f0 2024-04-26T20:11:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.103+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/3711355612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 0x564fb8509800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:40.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.103+0000 7f1fab400640 1 --2- 172.21.15.94:0/3711355612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb85092c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c009e50 tx=0x7f1f9c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbcf3c7995dd5b3a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:40.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.103+0000 7f1f9b400640 1 -- 172.21.15.94:0/3711355612 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c01f250 con 0x564fb84ea0f0 2024-04-26T20:11:40.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1f9b400640 1 -- 172.21.15.94:0/3711355612 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f9c004490 con 0x564fb84ea0f0 2024-04-26T20:11:40.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1f9b400640 1 -- 172.21.15.94:0/3711355612 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c010bb0 con 0x564fb84ea0f0 2024-04-26T20:11:40.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3711355612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 msgr2=0x564fb85092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:40.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3711355612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb85092c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c009e50 tx=0x7f1f9c005f90 comp rx=0 tx=0).stop 2024-04-26T20:11:40.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3711355612 shutdown_connections 2024-04-26T20:11:40.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3711355612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84efc50 0x564fb8509d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3711355612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84eaae0 0x564fb8509800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3711355612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb85092c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3711355612 >> 172.21.15.94:0/3711355612 conn(0x564fb847ed10 msgr2=0x564fb84e4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:40.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3711355612 shutdown_connections 2024-04-26T20:11:40.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.104+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3711355612 wait complete. 2024-04-26T20:11:40.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.105+0000 7f1fb4e07740 1 -- start start 2024-04-26T20:11:40.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.105+0000 7f1fb4e07740 1 Processor -- start 2024-04-26T20:11:40.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.105+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84fb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:40.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.105+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84fb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:40.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.105+0000 7f1fb4e07740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84f7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:40.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.105+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x564fb8501e70 con 0x564fb84ea0f0 2024-04-26T20:11:40.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.105+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564fb85017a0 con 0x564fb84eaae0 2024-04-26T20:11:40.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.105+0000 7f1fb4e07740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564fb8502540 con 0x564fb84efc50 2024-04-26T20:11:40.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.105+0000 7f1fb4e07740 10 client.0 _ll_register_callbacks cb 0x7f1f94005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:11:40.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fab400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84fb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fabe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84f7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fabe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84f7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:46114/0 (socket says 172.21.15.94:46114) 2024-04-26T20:11:40.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fa3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84fb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fabe00640 1 -- 172.21.15.94:0/3691070768 learned_addr learned my addr 172.21.15.94:0/3691070768 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:40.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fabe00640 1 -- 172.21.15.94:0/3691070768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 msgr2=0x564fb84fb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fabe00640 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84fb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fabe00640 1 -- 172.21.15.94:0/3691070768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 msgr2=0x564fb84fb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:40.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fabe00640 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84fb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:40.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fabe00640 1 -- 172.21.15.94:0/3691070768 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fb84f8340 con 0x564fb84efc50 2024-04-26T20:11:41.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.106+0000 7f1fa3e00640 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84fb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:41.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.107+0000 7f1fabe00640 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84f7c50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009ca0 tx=0x7f1fa40095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=857798c047bebbf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:41.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.107+0000 7f1f98c00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-04-26T20:11:41.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.107+0000 7f1f98c00640 1 -- 172.21.15.94:0/3691070768 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4004450 con 0x564fb84efc50 2024-04-26T20:11:41.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.107+0000 7f1f98c00640 1 -- 172.21.15.94:0/3691070768 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f1fa40045f0 con 0x564fb84efc50 2024-04-26T20:11:41.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.107+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3691070768 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f9c004490 con 0x564fb84efc50 2024-04-26T20:11:41.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.107+0000 7f1f98c00640 1 -- 172.21.15.94:0/3691070768 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1fa405f050 con 0x564fb84efc50 2024-04-26T20:11:41.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.107+0000 7f1fb4e07740 10 client.25144 fetch_fsmap learned FSMap version 24 2024-04-26T20:11:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.107+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3691070768 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564fb85677a0 con 0x564fb84efc50 2024-04-26T20:11:41.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.108+0000 7f1f98c00640 1 -- 172.21.15.94:0/3691070768 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f9c004490 con 0x564fb84efc50 2024-04-26T20:11:41.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.108+0000 7f1f98c00640 1 -- 172.21.15.94:0/3691070768 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1fa40306e0 con 0x564fb84efc50 2024-04-26T20:11:41.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.108+0000 7f1fb4e07740 10 client.25144 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:11:41.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.108+0000 7f1fb4e07740 10 client.25144 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T20:11:41.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.108+0000 7f1fb4e07740 20 client.25144 populate_metadata read hostname 'smithi094' 2024-04-26T20:11:41.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.108+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x564fb856c470 0x564fb85688b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:41.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.108+0000 7f1fb4e07740 4 client.25144 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:11:41.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.108+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3691070768 --> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] -- command(tid 0: {"prefix": "status"}) -- 0x564fb83d1860 con 0x564fb856c470 2024-04-26T20:11:41.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.108+0000 7f1fab400640 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x564fb856c470 0x564fb85688b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:41.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fab400640 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x564fb856c470 0x564fb85688b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=328683ddf05d6f4c server_cookie=689b1a7c9b651108 in_seq=0 out_seq=0 2024-04-26T20:11:41.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1f98c00640 10 client.25144 ms_handle_connect on v2:172.21.15.204:6838/3633033106 2024-04-26T20:11:41.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1f98c00640 1 -- 172.21.15.94:0/3691070768 <== mds.2 v2:172.21.15.204:6838/3633033106 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x564fb83d1860 con 0x564fb856c470 2024-04-26T20:11:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1f98c00640 10 client.25144 handle_command_reply: tid=0 2024-04-26T20:11:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 client.25144 shutdown 2024-04-26T20:11:41.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3691070768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 msgr2=0x564fb84f7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:41.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84f7c50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009ca0 tx=0x7f1fa40095c0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3691070768 shutdown_connections 2024-04-26T20:11:41.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x564fb856c470 0x564fb85688b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564fb84efc50 0x564fb84f7c50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1f98c00640 0 client.25144 ms_handle_reset on v2:172.21.15.204:6838/3633033106 2024-04-26T20:11:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564fb84eaae0 0x564fb84fb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 --2- 172.21.15.94:0/3691070768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x564fb84ea0f0 0x564fb84fb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3691070768 >> 172.21.15.94:0/3691070768 conn(0x564fb847ed10 msgr2=0x564fb84e8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:41.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.109+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3691070768 shutdown_connections 2024-04-26T20:11:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb4e07740 1 -- 172.21.15.94:0/3691070768 wait complete. 2024-04-26T20:11:41.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb4e07740 20 client.25144 trim_cache size 0 max 16384 2024-04-26T20:11:41.583 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-26T20:11:41.589 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T20:11:41.598 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 2, 2024-04-26T20:11:41.605 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24541, 2024-04-26T20:11:41.612 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-26T20:11:41.620 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-26T20:11:41.627 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-26T20:11:41.663 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 134.76408777200001, 2024-04-26T20:11:41.671 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 24, 2024-04-26T20:11:41.680 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 96, 2024-04-26T20:11:41.688 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 94, 2024-04-26T20:11:41.698 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 227.63029917 2024-04-26T20:11:41.708 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-26T20:11:41.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 -- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1f900a3c40 msgr2=0x7f1f900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:41.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1f900a3c40 0x7f1f900a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4005e30 tx=0x7f1fa4031af0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 -- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fac105d80 msgr2=0x7f1fac19b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fac105d80 0x7f1fac19b270 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e9f0 tx=0x7f1f9400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 -- 172.21.15.94:0/1512628029 shutdown_connections 2024-04-26T20:11:41.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1f900a3c40 0x7f1f900a6100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fac106740 0x7f1fac1a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fac105d80 0x7f1fac19b270 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 --2- 172.21.15.94:0/1512628029 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fac104b90 0x7f1fac19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:41.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.110+0000 7f1fb2000640 1 -- 172.21.15.94:0/1512628029 >> 172.21.15.94:0/1512628029 conn(0x7f1fac100120 msgr2=0x7f1fac109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:41.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.111+0000 7f1fb2000640 1 -- 172.21.15.94:0/1512628029 shutdown_connections 2024-04-26T20:11:42.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:38.111+0000 7f1fb2000640 1 -- 172.21.15.94:0/1512628029 wait complete. 2024-04-26T20:11:42.030 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:37 smithi094 ceph-mon[28050]: pgmap v429: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:42.039 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T20:11:42.052 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:40 smithi094 ceph-mon[28050]: pgmap v430: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:42.066 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:40 smithi204 ceph-mon[32618]: pgmap v430: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:42.077 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:37 smithi103 ceph-mon[32635]: pgmap v429: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:42.133 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:40 smithi103 ceph-mon[32635]: pgmap v430: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:42.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.273+0000 7f4c3a600640 1 -- 172.21.15.94:0/3281954688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c34108fb0 msgr2=0x7f4c341093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:42.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.273+0000 7f4c3a600640 1 --2- 172.21.15.94:0/3281954688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c34108fb0 0x7f4c341093d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009900 tx=0x7f4c2402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:11:42.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.273+0000 7f4c3a600640 1 -- 172.21.15.94:0/3281954688 shutdown_connections 2024-04-26T20:11:42.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.273+0000 7f4c3a600640 1 --2- 172.21.15.94:0/3281954688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c3410ab60 0x7f4c341113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:42.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.273+0000 7f4c3a600640 1 --2- 172.21.15.94:0/3281954688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c3410a1a0 0x7f4c3410a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:42.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.273+0000 7f4c3a600640 1 --2- 172.21.15.94:0/3281954688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c34108fb0 0x7f4c341093d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.273+0000 7f4c3a600640 1 -- 172.21.15.94:0/3281954688 >> 172.21.15.94:0/3281954688 conn(0x7f4c34077f40 msgr2=0x7f4c3407a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:42.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.273+0000 7f4c3a600640 1 -- 172.21.15.94:0/3281954688 shutdown_connections 2024-04-26T20:11:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.273+0000 7f4c3a600640 1 -- 172.21.15.94:0/3281954688 wait complete. 2024-04-26T20:11:42.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.274+0000 7f4c3a600640 1 Processor -- start 2024-04-26T20:11:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.274+0000 7f4c3a600640 1 -- start start 2024-04-26T20:11:42.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c3a600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c34108fb0 0x7f4c3419f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:42.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c3a600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c3410a1a0 0x7f4c3419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:42.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c3a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c3410ab60 0x7f4c341a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:42.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c3a600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4c34114b40 con 0x7f4c34108fb0 2024-04-26T20:11:42.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c3a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c341149c0 con 0x7f4c3410ab60 2024-04-26T20:11:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c3a600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c34114cc0 con 0x7f4c3410a1a0 2024-04-26T20:11:42.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c33400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c34108fb0 0x7f4c3419f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:42.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c3410a1a0 0x7f4c3419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:42.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c33400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c34108fb0 0x7f4c3419f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35754/0 (socket says 172.21.15.94:35754) 2024-04-26T20:11:42.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c33400640 1 -- 172.21.15.94:0/2200363964 learned_addr learned my addr 172.21.15.94:0/2200363964 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:42.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.275+0000 7f4c33e00640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c3410ab60 0x7f4c341a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:42.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c32a00640 1 -- 172.21.15.94:0/2200363964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c3410ab60 msgr2=0x7f4c341a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:42.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c32a00640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c3410ab60 0x7f4c341a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:42.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c32a00640 1 -- 172.21.15.94:0/2200363964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c34108fb0 msgr2=0x7f4c3419f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:42.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c32a00640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c34108fb0 0x7f4c3419f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:42.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c32a00640 1 -- 172.21.15.94:0/2200363964 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c341a6df0 con 0x7f4c3410a1a0 2024-04-26T20:11:42.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c32a00640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c3410a1a0 0x7f4c3419f710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000e9f0 tx=0x7f4c2000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:42.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c2be00640 1 -- 172.21.15.94:0/2200363964 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2000ccc0 con 0x7f4c3410a1a0 2024-04-26T20:11:42.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c2be00640 1 -- 172.21.15.94:0/2200363964 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c20005020 con 0x7f4c3410a1a0 2024-04-26T20:11:42.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c3a600640 1 -- 172.21.15.94:0/2200363964 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c341a70e0 con 0x7f4c3410a1a0 2024-04-26T20:11:42.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c2be00640 1 -- 172.21.15.94:0/2200363964 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2000ce60 con 0x7f4c3410a1a0 2024-04-26T20:11:42.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.276+0000 7f4c3a600640 1 -- 172.21.15.94:0/2200363964 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c341a7630 con 0x7f4c3410a1a0 2024-04-26T20:11:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c3d45e740 1 Processor -- start 2024-04-26T20:11:42.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c3d45e740 1 -- start start 2024-04-26T20:11:42.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae0f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:42.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae145e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:42.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae170e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:42.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556dbad73600 con 0x556dbae0fae0 2024-04-26T20:11:42.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556dbad10100 con 0x556dbae14c50 2024-04-26T20:11:42.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556dbad6f5c0 con 0x556dbae0f0f0 2024-04-26T20:11:42.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c33400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae145e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:42.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c2be00640 1 -- 172.21.15.94:0/2200363964 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4c200051d0 con 0x7f4c3410a1a0 2024-04-26T20:11:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c33400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae145e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35766/0 (socket says 172.21.15.94:35766) 2024-04-26T20:11:43.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c33400640 1 -- 172.21.15.94:0/3449325778 learned_addr learned my addr 172.21.15.94:0/3449325778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:43.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.280+0000 7f4c33e00640 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae0f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:43.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.281+0000 7f4c32a00640 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae170e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:43.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.281+0000 7f4c33400640 1 -- 172.21.15.94:0/3449325778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 msgr2=0x556dbae0f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:43.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.281+0000 7f4c33400640 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae0f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:43.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.281+0000 7f4c33400640 1 -- 172.21.15.94:0/3449325778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 msgr2=0x556dbae170e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:43.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.281+0000 7f4c33400640 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae170e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:43.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.281+0000 7f4c33400640 1 -- 172.21.15.94:0/3449325778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbaa852d0 con 0x556dbae0fae0 2024-04-26T20:11:43.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.281+0000 7f4c32a00640 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae170e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T20:11:43.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.281+0000 7f4c33400640 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae145e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402a9e0 tx=0x7f4c24004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da6c9d6ab812648 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:43.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c28c00640 1 -- 172.21.15.94:0/3449325778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24035030 con 0x556dbae0fae0 2024-04-26T20:11:43.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c28c00640 1 -- 172.21.15.94:0/3449325778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c24040070 con 0x556dbae0fae0 2024-04-26T20:11:43.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c28c00640 1 -- 172.21.15.94:0/3449325778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24031960 con 0x556dbae0fae0 2024-04-26T20:11:43.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c2be00640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4c180a3cc0 0x7f4c180a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:43.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c2be00640 1 -- 172.21.15.94:0/2200363964 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f4c200d0840 con 0x7f4c3410a1a0 2024-04-26T20:11:43.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c33e00640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4c180a3cc0 0x7f4c180a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:43.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c3d45e740 1 -- 172.21.15.94:0/3449325778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 msgr2=0x556dbae145e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:43.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae145e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402a9e0 tx=0x7f4c24004040 comp rx=0 tx=0).stop 2024-04-26T20:11:43.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c3d45e740 1 -- 172.21.15.94:0/3449325778 shutdown_connections 2024-04-26T20:11:43.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae170e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:43.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.282+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae145e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.283+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/3449325778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae0f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:43.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.283+0000 7f4c3d45e740 1 -- 172.21.15.94:0/3449325778 >> 172.21.15.94:0/3449325778 conn(0x556dbada3d10 msgr2=0x556dbae0ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:43.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.283+0000 7f4c3d45e740 1 -- 172.21.15.94:0/3449325778 shutdown_connections 2024-04-26T20:11:43.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.283+0000 7f4c3d45e740 1 -- 172.21.15.94:0/3449325778 wait complete. 2024-04-26T20:11:43.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.283+0000 7f4c33e00640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4c180a3cc0 0x7f4c180a6180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x556dbae10340 tx=0x7f4c2c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:43.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.283+0000 7f4c3d45e740 1 -- start start 2024-04-26T20:11:43.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.283+0000 7f4c3d45e740 1 Processor -- start 2024-04-26T20:11:43.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:43.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae0fae0 0x556dbae19a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:43.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae14c50 0x556dbae20a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:43.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556dbae17ba0 con 0x556dbae14c50 2024-04-26T20:11:43.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556dbacfaf70 con 0x556dbae0fae0 2024-04-26T20:11:43.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556dbae17d20 con 0x556dbae0f0f0 2024-04-26T20:11:43.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c3d45e740 10 client.0 _ll_register_callbacks cb 0x556dbad69480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:11:43.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae14c50 0x556dbae20a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:43.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae14c50 0x556dbae20a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35780/0 (socket says 172.21.15.94:35780) 2024-04-26T20:11:43.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33e00640 1 -- 172.21.15.94:0/1089297826 learned_addr learned my addr 172.21.15.94:0/1089297826 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:43.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c32a00640 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae0fae0 0x556dbae19a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:43.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33400640 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:43.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33e00640 1 -- 172.21.15.94:0/1089297826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 msgr2=0x556dbae19510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:43.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33e00640 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:43.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33e00640 1 -- 172.21.15.94:0/1089297826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae0fae0 msgr2=0x556dbae19a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:43.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33e00640 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae0fae0 0x556dbae19a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:43.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33e00640 1 -- 172.21.15.94:0/1089297826 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbab4f6b0 con 0x556dbae14c50 2024-04-26T20:11:43.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c33400640 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:11:43.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.284+0000 7f4c32a00640 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae0fae0 0x556dbae19a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:43.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.285+0000 7f4c33e00640 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae14c50 0x556dbae20a90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009fd0 tx=0x7f4c2c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb464a8f1ab57c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:43.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.285+0000 7f4c1d600640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-26T20:11:43.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.285+0000 7f4c1d600640 1 -- 172.21.15.94:0/1089297826 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00f7d0 con 0x556dbae14c50 2024-04-26T20:11:44.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.285+0000 7f4c1d600640 1 -- 172.21.15.94:0/1089297826 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f4c2c00f970 con 0x556dbae14c50 2024-04-26T20:11:44.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.285+0000 7f4c3d45e740 1 -- 172.21.15.94:0/1089297826 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c24040070 con 0x556dbae14c50 2024-04-26T20:11:44.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.285+0000 7f4c1d600640 1 -- 172.21.15.94:0/1089297826 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4c2c00fb70 con 0x556dbae14c50 2024-04-26T20:11:44.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.285+0000 7f4c3d45e740 10 client.15063 fetch_fsmap learned FSMap version 24 2024-04-26T20:11:44.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.285+0000 7f4c3d45e740 1 -- 172.21.15.94:0/1089297826 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556dbae23ab0 con 0x556dbae14c50 2024-04-26T20:11:44.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.286+0000 7f4c1d600640 1 -- 172.21.15.94:0/1089297826 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c24040070 con 0x556dbae14c50 2024-04-26T20:11:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.286+0000 7f4c1d600640 1 -- 172.21.15.94:0/1089297826 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4c2c027070 con 0x556dbae14c50 2024-04-26T20:11:44.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.286+0000 7f4c3d45e740 10 client.15063 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:11:44.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.286+0000 7f4c3d45e740 10 client.15063 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T20:11:44.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.286+0000 7f4c3d45e740 20 client.15063 populate_metadata read hostname 'smithi094' 2024-04-26T20:11:44.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.286+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x556dbae261a0 0x556dbae27610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:44.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.286+0000 7f4c3d45e740 4 client.15063 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:11:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.286+0000 7f4c3d45e740 1 -- 172.21.15.94:0/1089297826 --> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dbad81210 con 0x556dbae261a0 2024-04-26T20:11:44.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.286+0000 7f4c33400640 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x556dbae261a0 0x556dbae27610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:44.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.287+0000 7f4c33400640 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x556dbae261a0 0x556dbae27610 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b48fe426d35aaead server_cookie=9f968234a6253190 in_seq=0 out_seq=0 2024-04-26T20:11:44.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.287+0000 7f4c1d600640 10 client.15063 ms_handle_connect on v2:172.21.15.94:6840/989331046 2024-04-26T20:11:44.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c1d600640 1 -- 172.21.15.94:0/1089297826 <== mds.1 v2:172.21.15.94:6840/989331046 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556dbad81210 con 0x556dbae261a0 2024-04-26T20:11:44.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c1d600640 10 client.15063 handle_command_reply: tid=0 2024-04-26T20:11:44.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c3d45e740 1 client.15063 shutdown 2024-04-26T20:11:44.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c3d45e740 1 -- 172.21.15.94:0/1089297826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae14c50 msgr2=0x556dbae20a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae14c50 0x556dbae20a90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009fd0 tx=0x7f4c2c004ab0 comp rx=0 tx=0).stop 2024-04-26T20:11:44.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c3d45e740 1 -- 172.21.15.94:0/1089297826 shutdown_connections 2024-04-26T20:11:44.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x556dbae261a0 0x556dbae27610 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:44.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae14c50 0x556dbae20a90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:44.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae0fae0 0x556dbae19a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:44.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/1089297826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c1d600640 0 client.15063 ms_handle_reset on v2:172.21.15.94:6840/989331046 2024-04-26T20:11:44.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.288+0000 7f4c3d45e740 1 -- 172.21.15.94:0/1089297826 >> 172.21.15.94:0/1089297826 conn(0x556dbada3d10 msgr2=0x556dbae09eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:44.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.289+0000 7f4c3d45e740 1 -- 172.21.15.94:0/1089297826 shutdown_connections 2024-04-26T20:11:44.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.289+0000 7f4c3d45e740 1 -- 172.21.15.94:0/1089297826 wait complete. 2024-04-26T20:11:44.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.289+0000 7f4c3d45e740 20 client.15063 trim_cache size 0 max 16384 2024-04-26T20:11:44.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.308+0000 7f4c3d45e740 1 Processor -- start 2024-04-26T20:11:44.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.308+0000 7f4c3d45e740 1 -- start start 2024-04-26T20:11:44.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.308+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:44.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.308+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0fae0 0x556dbae19a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:44.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.308+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae209b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:44.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.308+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556dbae25950 con 0x556dbae0f0f0 2024-04-26T20:11:44.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.308+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556dbae26020 con 0x556dbae14c50 2024-04-26T20:11:44.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.308+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556dbae266f0 con 0x556dbae0fae0 2024-04-26T20:11:44.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:44.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35782/0 (socket says 172.21.15.94:35782) 2024-04-26T20:11:44.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33400640 1 -- 172.21.15.94:0/959923925 learned_addr learned my addr 172.21.15.94:0/959923925 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:44.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c32a00640 1 --2- 172.21.15.94:0/959923925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0fae0 0x556dbae19a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:44.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33e00640 1 --2- 172.21.15.94:0/959923925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:44.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33400640 1 -- 172.21.15.94:0/959923925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0fae0 msgr2=0x556dbae19a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:44.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33400640 1 --2- 172.21.15.94:0/959923925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0fae0 0x556dbae19a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:44.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33400640 1 -- 172.21.15.94:0/959923925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 msgr2=0x556dbae209b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:44.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33400640 1 --2- 172.21.15.94:0/959923925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae209b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33400640 1 -- 172.21.15.94:0/959923925 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbae10370 con 0x556dbae0f0f0 2024-04-26T20:11:44.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.309+0000 7f4c33400640 1 --2- 172.21.15.94:0/959923925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402a910 tx=0x7f4c24047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4204df1d93d4b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c1fe00640 1 -- 172.21.15.94:0/959923925 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24035040 con 0x556dbae0f0f0 2024-04-26T20:11:44.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c1fe00640 1 -- 172.21.15.94:0/959923925 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c24040070 con 0x556dbae0f0f0 2024-04-26T20:11:45.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c1fe00640 1 -- 172.21.15.94:0/959923925 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24031c20 con 0x556dbae0f0f0 2024-04-26T20:11:45.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c3d45e740 1 -- 172.21.15.94:0/959923925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0f0f0 msgr2=0x556dbae19510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:45.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/959923925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4c2402a910 tx=0x7f4c24047040 comp rx=0 tx=0).stop 2024-04-26T20:11:45.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c3d45e740 1 -- 172.21.15.94:0/959923925 shutdown_connections 2024-04-26T20:11:45.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/959923925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae209b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:45.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/959923925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0fae0 0x556dbae19a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:45.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/959923925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0f0f0 0x556dbae19510 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:45.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c3d45e740 1 -- 172.21.15.94:0/959923925 >> 172.21.15.94:0/959923925 conn(0x556dbada3d10 msgr2=0x556dbae180c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:45.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.310+0000 7f4c3d45e740 1 -- 172.21.15.94:0/959923925 shutdown_connections 2024-04-26T20:11:45.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.311+0000 7f4c3d45e740 1 -- 172.21.15.94:0/959923925 wait complete. 2024-04-26T20:11:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.311+0000 7f4c3d45e740 1 -- start start 2024-04-26T20:11:45.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.311+0000 7f4c3d45e740 1 Processor -- start 2024-04-26T20:11:45.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.311+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae1c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.311+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae1c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:45.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.311+0000 7f4c3d45e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae1f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:45.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.311+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x556dbae27490 con 0x556dbae0fae0 2024-04-26T20:11:45.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.311+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556dbae26dc0 con 0x556dbae14c50 2024-04-26T20:11:45.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.311+0000 7f4c3d45e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556dbae27b60 con 0x556dbae0f0f0 2024-04-26T20:11:45.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c3d45e740 10 client.0 _ll_register_callbacks cb 0x7f4c2c009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:11:45.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae1c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:45.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c32a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae1c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35798/0 (socket says 172.21.15.94:35798) 2024-04-26T20:11:45.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c32a00640 1 -- 172.21.15.94:0/2254416762 learned_addr learned my addr 172.21.15.94:0/2254416762 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:45.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c33400640 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae1c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:45.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c33e00640 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae1f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:45.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c32a00640 1 -- 172.21.15.94:0/2254416762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 msgr2=0x556dbae1c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:45.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c32a00640 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae1c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:45.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c32a00640 1 -- 172.21.15.94:0/2254416762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 msgr2=0x556dbae1f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c32a00640 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae1f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:45.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c32a00640 1 -- 172.21.15.94:0/2254416762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbae1feb0 con 0x556dbae0fae0 2024-04-26T20:11:45.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c32a00640 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae1c8a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4c200cde50 tx=0x7f4c2009c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe98d4c0cd101a14 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:45.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.312+0000 7f4c1d600640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-26T20:11:45.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c1d600640 1 -- 172.21.15.94:0/2254416762 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2001c040 con 0x556dbae0fae0 2024-04-26T20:11:45.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c1d600640 1 -- 172.21.15.94:0/2254416762 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f4c200a4dd0 con 0x556dbae0fae0 2024-04-26T20:11:45.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c3d45e740 1 -- 172.21.15.94:0/2254416762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c24040070 con 0x556dbae0fae0 2024-04-26T20:11:45.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c1d600640 1 -- 172.21.15.94:0/2254416762 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4c200a09a0 con 0x556dbae0fae0 2024-04-26T20:11:45.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c3d45e740 10 client.15069 fetch_fsmap learned FSMap version 24 2024-04-26T20:11:45.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c3d45e740 1 -- 172.21.15.94:0/2254416762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556dbae8c350 con 0x556dbae0fae0 2024-04-26T20:11:45.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c1d600640 1 -- 172.21.15.94:0/2254416762 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c24040070 con 0x556dbae0fae0 2024-04-26T20:11:45.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c1d600640 1 -- 172.21.15.94:0/2254416762 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4c20098340 con 0x556dbae0fae0 2024-04-26T20:11:45.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c3d45e740 10 client.15069 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:11:45.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c3d45e740 10 client.15069 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T20:11:45.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c3d45e740 20 client.15069 populate_metadata read hostname 'smithi094' 2024-04-26T20:11:46.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.313+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x556dbae91020 0x556dbae8fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:46.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.314+0000 7f4c3d45e740 4 client.15069 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:11:46.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.314+0000 7f4c3d45e740 1 -- 172.21.15.94:0/2254416762 --> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] -- command(tid 0: {"prefix": "status"}) -- 0x556dbad7b080 con 0x556dbae91020 2024-04-26T20:11:46.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.314+0000 7f4c33400640 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x556dbae91020 0x556dbae8fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.314+0000 7f4c33400640 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x556dbae91020 0x556dbae8fd30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b39a45a75c3f9e56 server_cookie=650bba3bb5b1f797 in_seq=0 out_seq=0 2024-04-26T20:11:46.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.314+0000 7f4c1d600640 10 client.15069 ms_handle_connect on v2:172.21.15.94:6840/989331046 2024-04-26T20:11:46.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c1d600640 1 -- 172.21.15.94:0/2254416762 <== mds.1 v2:172.21.15.94:6840/989331046 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x556dbad7b080 con 0x556dbae91020 2024-04-26T20:11:46.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c1d600640 10 client.15069 handle_command_reply: tid=0 2024-04-26T20:11:46.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 client.15069 shutdown 2024-04-26T20:11:46.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 -- 172.21.15.94:0/2254416762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 msgr2=0x556dbae1c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:46.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae1c8a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4c200cde50 tx=0x7f4c2009c1f0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 -- 172.21.15.94:0/2254416762 shutdown_connections 2024-04-26T20:11:46.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x556dbae91020 0x556dbae8fd30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556dbae14c50 0x556dbae1f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556dbae0fae0 0x556dbae1c8a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c1d600640 0 client.15069 ms_handle_reset on v2:172.21.15.94:6840/989331046 2024-04-26T20:11:46.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 --2- 172.21.15.94:0/2254416762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556dbae0f0f0 0x556dbae1c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 -- 172.21.15.94:0/2254416762 >> 172.21.15.94:0/2254416762 conn(0x556dbada3d10 msgr2=0x556dbae10f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:46.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 -- 172.21.15.94:0/2254416762 shutdown_connections 2024-04-26T20:11:46.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 1 -- 172.21.15.94:0/2254416762 wait complete. 2024-04-26T20:11:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.315+0000 7f4c3d45e740 20 client.15069 trim_cache size 0 max 16384 2024-04-26T20:11:46.388 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-26T20:11:46.394 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T20:11:46.401 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 1, 2024-04-26T20:11:46.408 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24562, 2024-04-26T20:11:46.416 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-26T20:11:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-26T20:11:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-26T20:11:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 139.983623228, 2024-04-26T20:11:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 24, 2024-04-26T20:11:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 96, 2024-04-26T20:11:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 94, 2024-04-26T20:11:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 223.406569566 2024-04-26T20:11:46.607 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-26T20:11:46.622 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:42 smithi204 ceph-mon[32618]: pgmap v431: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:46.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:44 smithi204 ceph-mon[32618]: pgmap v432: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:46.685 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:42 smithi103 ceph-mon[32635]: pgmap v431: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:46.691 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:44 smithi103 ceph-mon[32635]: pgmap v432: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:46.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 -- 172.21.15.94:0/2200363964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4c180a3cc0 msgr2=0x7f4c180a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:46.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4c180a3cc0 0x7f4c180a6180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x556dbae10340 tx=0x7f4c2c009500 comp rx=0 tx=0).stop 2024-04-26T20:11:46.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 -- 172.21.15.94:0/2200363964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c3410a1a0 msgr2=0x7f4c3419f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c3410a1a0 0x7f4c3419f710 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000e9f0 tx=0x7f4c2000edf0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 -- 172.21.15.94:0/2200363964 shutdown_connections 2024-04-26T20:11:46.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4c180a3cc0 0x7f4c180a6180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c3410ab60 0x7f4c341a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c3410a1a0 0x7f4c3419f710 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 --2- 172.21.15.94:0/2200363964 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c34108fb0 0x7f4c3419f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:46.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 -- 172.21.15.94:0/2200363964 >> 172.21.15.94:0/2200363964 conn(0x7f4c34077f40 msgr2=0x7f4c3410d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:46.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 -- 172.21.15.94:0/2200363964 shutdown_connections 2024-04-26T20:11:46.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:42.316+0000 7f4c3a600640 1 -- 172.21.15.94:0/2200363964 wait complete. 2024-04-26T20:11:46.867 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T20:11:46.910 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:42 smithi094 ceph-mon[28050]: pgmap v431: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:46.918 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:44 smithi094 ceph-mon[28050]: pgmap v432: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:46.924 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:46 smithi094 ceph-mon[28050]: pgmap v433: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:47.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:46 smithi103 ceph-mon[32635]: pgmap v433: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:47.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.129+0000 7f2691600640 1 -- 172.21.15.94:0/3160125076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f268c104520 msgr2=0x7f268c10adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:47.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.129+0000 7f2691600640 1 --2- 172.21.15.94:0/3160125076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f268c104520 0x7f268c10adb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2684009ca0 tx=0x7f268402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:11:47.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.129+0000 7f2691600640 1 -- 172.21.15.94:0/3160125076 shutdown_connections 2024-04-26T20:11:47.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.129+0000 7f2691600640 1 --2- 172.21.15.94:0/3160125076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f268c104520 0x7f268c10adb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:47.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.129+0000 7f2691600640 1 --2- 172.21.15.94:0/3160125076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f268c103b60 0x7f268c103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:47.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.129+0000 7f2691600640 1 --2- 172.21.15.94:0/3160125076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f268c102970 0x7f268c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:47.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.129+0000 7f2691600640 1 -- 172.21.15.94:0/3160125076 >> 172.21.15.94:0/3160125076 conn(0x7f268c0fdee0 msgr2=0x7f268c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:47.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.129+0000 7f2691600640 1 -- 172.21.15.94:0/3160125076 shutdown_connections 2024-04-26T20:11:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.129+0000 7f2691600640 1 -- 172.21.15.94:0/3160125076 wait complete. 2024-04-26T20:11:47.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:46 smithi204 ceph-mon[32618]: pgmap v433: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:47.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.130+0000 7f2691600640 1 Processor -- start 2024-04-26T20:11:47.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.130+0000 7f2691600640 1 -- start start 2024-04-26T20:11:47.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.130+0000 7f2691600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f268c102970 0x7f268c19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:47.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.130+0000 7f2691600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f268c103b60 0x7f268c19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:47.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.130+0000 7f2691600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f268c104520 0x7f268c1a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:47.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.130+0000 7f2691600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10e4f0 con 0x7f268c103b60 2024-04-26T20:11:47.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.130+0000 7f2691600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10e370 con 0x7f268c104520 2024-04-26T20:11:47.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.130+0000 7f2691600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10e670 con 0x7f268c102970 2024-04-26T20:11:47.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f268c102970 0x7f268c19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:47.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f268c102970 0x7f268c19ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:34188/0 (socket says 172.21.15.94:34188) 2024-04-26T20:11:47.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268aa00640 1 -- 172.21.15.94:0/4162567648 learned_addr learned my addr 172.21.15.94:0/4162567648 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:47.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268a000640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f268c103b60 0x7f268c19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:47.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268b400640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f268c104520 0x7f268c1a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:47.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268aa00640 1 -- 172.21.15.94:0/4162567648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f268c104520 msgr2=0x7f268c1a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:47.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268aa00640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f268c104520 0x7f268c1a2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268aa00640 1 -- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f268c103b60 msgr2=0x7f268c19b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:47.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268aa00640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f268c103b60 0x7f268c19b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:47.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268aa00640 1 -- 172.21.15.94:0/4162567648 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c1a2970 con 0x7f268c102970 2024-04-26T20:11:47.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.131+0000 7f268aa00640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f268c102970 0x7f268c19ad50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e9c0 tx=0x7f267c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:47.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.132+0000 7f2683400640 1 -- 172.21.15.94:0/4162567648 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c00cd80 con 0x7f268c102970 2024-04-26T20:11:47.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.132+0000 7f2683400640 1 -- 172.21.15.94:0/4162567648 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f267c0035f0 con 0x7f268c102970 2024-04-26T20:11:47.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.132+0000 7f2691600640 1 -- 172.21.15.94:0/4162567648 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c1a2c60 con 0x7f268c102970 2024-04-26T20:11:47.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.132+0000 7f2683400640 1 -- 172.21.15.94:0/4162567648 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c010690 con 0x7f268c102970 2024-04-26T20:11:47.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.132+0000 7f2691600640 1 -- 172.21.15.94:0/4162567648 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c1a31b0 con 0x7f268c102970 2024-04-26T20:11:47.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f269439a740 1 Processor -- start 2024-04-26T20:11:47.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f269439a740 1 -- start start 2024-04-26T20:11:47.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f2683400640 1 -- 172.21.15.94:0/4162567648 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f267c0037a0 con 0x7f268c102970 2024-04-26T20:11:47.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f269439a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 0x563e91e24510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:47.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f269439a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 0x563e91e295e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:47.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f269439a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e29c50 0x563e91e2c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:47.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f269439a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563e91d88600 con 0x563e91e29c50 2024-04-26T20:11:47.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f269439a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563e91d25100 con 0x563e91e24ae0 2024-04-26T20:11:47.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f269439a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x563e91d845c0 con 0x563e91e240f0 2024-04-26T20:11:47.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f268a000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e29c50 0x563e91e2c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:47.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f268b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 0x563e91e24510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:47.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f268a000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e29c50 0x563e91e2c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35826/0 (socket says 172.21.15.94:35826) 2024-04-26T20:11:47.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f268a000640 1 -- 172.21.15.94:0/4008732129 learned_addr learned my addr 172.21.15.94:0/4008732129 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:48.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.135+0000 7f268aa00640 1 --2- 172.21.15.94:0/4008732129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 0x563e91e295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:48.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.136+0000 7f268b400640 1 -- 172.21.15.94:0/4008732129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 msgr2=0x563e91e295e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:48.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.136+0000 7f268b400640 1 --2- 172.21.15.94:0/4008732129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 0x563e91e295e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.136+0000 7f268b400640 1 -- 172.21.15.94:0/4008732129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e29c50 msgr2=0x563e91e2c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:48.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.136+0000 7f268b400640 1 --2- 172.21.15.94:0/4008732129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e29c50 0x563e91e2c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.136+0000 7f268b400640 1 -- 172.21.15.94:0/4008732129 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e91a9a2d0 con 0x563e91e240f0 2024-04-26T20:11:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.136+0000 7f268b400640 1 --2- 172.21.15.94:0/4008732129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 0x563e91e24510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f268402ade0 tx=0x7f26840093c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d0abd3463edf4b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:48.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f2677400640 1 -- 172.21.15.94:0/4008732129 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684031650 con 0x563e91e240f0 2024-04-26T20:11:48.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f2677400640 1 -- 172.21.15.94:0/4008732129 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2684036030 con 0x563e91e240f0 2024-04-26T20:11:48.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f2677400640 1 -- 172.21.15.94:0/4008732129 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26840045c0 con 0x563e91e240f0 2024-04-26T20:11:48.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f2683400640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26700a3c40 0x7f26700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:48.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f2683400640 1 -- 172.21.15.94:0/4162567648 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f267c0d0d20 con 0x7f268c102970 2024-04-26T20:11:48.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f268aa00640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26700a3c40 0x7f26700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:48.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f269439a740 1 -- 172.21.15.94:0/4008732129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 msgr2=0x563e91e24510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:48.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f269439a740 1 --2- 172.21.15.94:0/4008732129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 0x563e91e24510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f268402ade0 tx=0x7f26840093c0 comp rx=0 tx=0).stop 2024-04-26T20:11:48.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f269439a740 1 -- 172.21.15.94:0/4008732129 shutdown_connections 2024-04-26T20:11:48.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f269439a740 1 --2- 172.21.15.94:0/4008732129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e29c50 0x563e91e2c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:48.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f269439a740 1 --2- 172.21.15.94:0/4008732129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 0x563e91e295e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:48.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f269439a740 1 --2- 172.21.15.94:0/4008732129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 0x563e91e24510 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:48.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.137+0000 7f269439a740 1 -- 172.21.15.94:0/4008732129 >> 172.21.15.94:0/4008732129 conn(0x563e91db8d10 msgr2=0x563e91e1fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:48.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.138+0000 7f269439a740 1 -- 172.21.15.94:0/4008732129 shutdown_connections 2024-04-26T20:11:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.138+0000 7f269439a740 1 -- 172.21.15.94:0/4008732129 wait complete. 2024-04-26T20:11:48.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.138+0000 7f268aa00640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26700a3c40 0x7f26700a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x563e91e24ee0 tx=0x7f267c0d47d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:48.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.138+0000 7f269439a740 1 -- start start 2024-04-26T20:11:48.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.138+0000 7f269439a740 1 Processor -- start 2024-04-26T20:11:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f269439a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e2e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:48.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f269439a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 0x563e91e2e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f269439a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e29c50 0x563e91e358f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:48.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f269439a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563e91e2ca20 con 0x563e91e240f0 2024-04-26T20:11:48.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f269439a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563e91d0ff70 con 0x563e91e24ae0 2024-04-26T20:11:48.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f269439a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x563e91e2cba0 con 0x563e91e29c50 2024-04-26T20:11:48.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f269439a740 10 client.0 _ll_register_callbacks cb 0x563e91d7e480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:11:48.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e2e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:48.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e2e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35828/0 (socket says 172.21.15.94:35828) 2024-04-26T20:11:48.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268b400640 1 -- 172.21.15.94:0/3564415998 learned_addr learned my addr 172.21.15.94:0/3564415998 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:48.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268a000640 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 0x563e91e2e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268b400640 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e29c50 0x563e91e358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268a000640 1 -- 172.21.15.94:0/3564415998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e29c50 msgr2=0x563e91e358f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:48.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268a000640 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e29c50 0x563e91e358f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:48.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268a000640 1 -- 172.21.15.94:0/3564415998 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 msgr2=0x563e91e2e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:48.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268a000640 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e2e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:48.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268a000640 1 -- 172.21.15.94:0/3564415998 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e91b646b0 con 0x563e91e24ae0 2024-04-26T20:11:48.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.139+0000 7f268b400640 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e2e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:48.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.140+0000 7f268a000640 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 0x563e91e2e8b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f26780126d0 tx=0x7f2678012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb5bf8f1d8fd2d0a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:48.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.140+0000 7f266be00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-26T20:11:48.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.140+0000 7f266be00640 1 -- 172.21.15.94:0/3564415998 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678004030 con 0x563e91e24ae0 2024-04-26T20:11:48.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.140+0000 7f269439a740 1 -- 172.21.15.94:0/3564415998 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2684036030 con 0x563e91e24ae0 2024-04-26T20:11:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.140+0000 7f266be00640 1 -- 172.21.15.94:0/3564415998 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f26780041d0 con 0x563e91e24ae0 2024-04-26T20:11:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.140+0000 7f266be00640 1 -- 172.21.15.94:0/3564415998 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2678025050 con 0x563e91e24ae0 2024-04-26T20:11:48.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.140+0000 7f269439a740 10 client.25184 fetch_fsmap learned FSMap version 24 2024-04-26T20:11:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.140+0000 7f269439a740 1 -- 172.21.15.94:0/3564415998 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563e91e38990 con 0x563e91e24ae0 2024-04-26T20:11:49.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.141+0000 7f266be00640 1 -- 172.21.15.94:0/3564415998 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2684036030 con 0x563e91e24ae0 2024-04-26T20:11:49.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.141+0000 7f266be00640 1 -- 172.21.15.94:0/3564415998 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2678003900 con 0x563e91e24ae0 2024-04-26T20:11:49.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.141+0000 7f269439a740 10 client.25184 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:11:49.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.141+0000 7f269439a740 10 client.25184 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:11:49.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.141+0000 7f269439a740 20 client.25184 populate_metadata read hostname 'smithi094' 2024-04-26T20:11:49.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.141+0000 7f269439a740 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x563e91e3aff0 0x563e91e3c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:49.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.141+0000 7f269439a740 4 client.25184 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:11:49.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.141+0000 7f269439a740 1 -- 172.21.15.94:0/3564415998 --> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e91d96210 con 0x563e91e3aff0 2024-04-26T20:11:49.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.141+0000 7f268b400640 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x563e91e3aff0 0x563e91e3c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:49.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.142+0000 7f268b400640 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x563e91e3aff0 0x563e91e3c460 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b7c02a0a731e5db server_cookie=408b710b8328d8a1 in_seq=0 out_seq=0 2024-04-26T20:11:49.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.142+0000 7f266be00640 10 client.25184 ms_handle_connect on v2:172.21.15.103:6838/1048742250 2024-04-26T20:11:49.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f266be00640 1 -- 172.21.15.94:0/3564415998 <== mds.0 v2:172.21.15.103:6838/1048742250 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e91d96210 con 0x563e91e3aff0 2024-04-26T20:11:49.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f266be00640 10 client.25184 handle_command_reply: tid=0 2024-04-26T20:11:49.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 client.25184 shutdown 2024-04-26T20:11:49.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 -- 172.21.15.94:0/3564415998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 msgr2=0x563e91e2e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:49.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 0x563e91e2e8b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f26780126d0 tx=0x7f2678012b90 comp rx=0 tx=0).stop 2024-04-26T20:11:49.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 -- 172.21.15.94:0/3564415998 shutdown_connections 2024-04-26T20:11:49.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x563e91e3aff0 0x563e91e3c460 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e29c50 0x563e91e358f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e24ae0 0x563e91e2e8b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 --2- 172.21.15.94:0/3564415998 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e2e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 -- 172.21.15.94:0/3564415998 >> 172.21.15.94:0/3564415998 conn(0x563e91db8d10 msgr2=0x563e91e1eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:49.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f266be00640 0 client.25184 ms_handle_reset on v2:172.21.15.103:6838/1048742250 2024-04-26T20:11:49.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 -- 172.21.15.94:0/3564415998 shutdown_connections 2024-04-26T20:11:49.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 1 -- 172.21.15.94:0/3564415998 wait complete. 2024-04-26T20:11:49.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.143+0000 7f269439a740 20 client.25184 trim_cache size 0 max 16384 2024-04-26T20:11:49.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.163+0000 7f269439a740 1 Processor -- start 2024-04-26T20:11:49.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.163+0000 7f269439a740 1 -- start start 2024-04-26T20:11:49.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f269439a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e25580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:49.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f269439a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e24ae0 0x563e91e358b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:49.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f269439a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 0x563e91e2df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:49.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f269439a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563e91e3a7a0 con 0x563e91e240f0 2024-04-26T20:11:49.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f269439a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563e91e3ae70 con 0x563e91e29c50 2024-04-26T20:11:49.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f269439a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x563e91e3b540 con 0x563e91e24ae0 2024-04-26T20:11:49.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e25580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:49.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268b400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e25580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35840/0 (socket says 172.21.15.94:35840) 2024-04-26T20:11:49.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268b400640 1 -- 172.21.15.94:0/3256091502 learned_addr learned my addr 172.21.15.94:0/3256091502 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:49.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268a000640 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e24ae0 0x563e91e358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:49.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268b400640 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 0x563e91e2df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:49.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268a000640 1 -- 172.21.15.94:0/3256091502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 msgr2=0x563e91e2df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:49.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268a000640 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 0x563e91e2df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268a000640 1 -- 172.21.15.94:0/3256091502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 msgr2=0x563e91e25580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:49.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268a000640 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e25580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268a000640 1 -- 172.21.15.94:0/3256091502 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e91e291d0 con 0x563e91e24ae0 2024-04-26T20:11:49.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.164+0000 7f268b400640 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e25580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:49.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.165+0000 7f268a000640 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e24ae0 0x563e91e358b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2678009950 tx=0x7f26780106c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a0261c2fcb7ca2b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:49.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.165+0000 7f2676a00640 1 -- 172.21.15.94:0/3256091502 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678022040 con 0x563e91e24ae0 2024-04-26T20:11:49.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.165+0000 7f2676a00640 1 -- 172.21.15.94:0/3256091502 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2678005070 con 0x563e91e24ae0 2024-04-26T20:11:49.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.165+0000 7f2676a00640 1 -- 172.21.15.94:0/3256091502 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678010a30 con 0x563e91e24ae0 2024-04-26T20:11:49.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 -- 172.21.15.94:0/3256091502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e24ae0 msgr2=0x563e91e358b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e24ae0 0x563e91e358b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2678009950 tx=0x7f26780106c0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 -- 172.21.15.94:0/3256091502 shutdown_connections 2024-04-26T20:11:49.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 0x563e91e2df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e24ae0 0x563e91e358b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 --2- 172.21.15.94:0/3256091502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e240f0 0x563e91e25580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:49.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 -- 172.21.15.94:0/3256091502 >> 172.21.15.94:0/3256091502 conn(0x563e91db8d10 msgr2=0x563e91e25ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:49.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 -- 172.21.15.94:0/3256091502 shutdown_connections 2024-04-26T20:11:49.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 -- 172.21.15.94:0/3256091502 wait complete. 2024-04-26T20:11:49.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 -- start start 2024-04-26T20:11:49.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.166+0000 7f269439a740 1 Processor -- start 2024-04-26T20:11:49.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f269439a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 0x563e91e31550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:49.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f269439a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e24ae0 0x563e91e31a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:49.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f269439a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 0x563e91e34ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:49.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f269439a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x563e91e3c2e0 con 0x563e91e24ae0 2024-04-26T20:11:49.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f269439a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563e91e3bc10 con 0x563e91e29c50 2024-04-26T20:11:50.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f269439a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x563e91e3c9b0 con 0x563e91e240f0 2024-04-26T20:11:50.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f269439a740 10 client.0 _ll_register_callbacks cb 0x7f2678009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:11:50.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f268a000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e24ae0 0x563e91e31a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:50.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f268a000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e24ae0 0x563e91e31a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35854/0 (socket says 172.21.15.94:35854) 2024-04-26T20:11:50.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f268a000640 1 -- 172.21.15.94:0/329770033 learned_addr learned my addr 172.21.15.94:0/329770033 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:50.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f268b400640 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 0x563e91e31550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f268b400640 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 0x563e91e34ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f268a000640 1 -- 172.21.15.94:0/329770033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 msgr2=0x563e91e31550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:50.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f268a000640 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 0x563e91e31550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:50.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f268a000640 1 -- 172.21.15.94:0/329770033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 msgr2=0x563e91e34ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:11:50.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.167+0000 7f268a000640 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 0x563e91e34ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:50.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f268a000640 1 -- 172.21.15.94:0/329770033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e91e350a0 con 0x563e91e24ae0 2024-04-26T20:11:50.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f268b400640 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 0x563e91e34ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:11:50.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f268a000640 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e24ae0 0x563e91e31a90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2678012420 tx=0x7f2678004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6068e9c30df40681 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f266be00640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-26T20:11:50.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f266be00640 1 -- 172.21.15.94:0/329770033 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26780038c0 con 0x563e91e24ae0 2024-04-26T20:11:50.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f266be00640 1 -- 172.21.15.94:0/329770033 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f2678005220 con 0x563e91e24ae0 2024-04-26T20:11:50.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f269439a740 1 -- 172.21.15.94:0/329770033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2678005070 con 0x563e91e24ae0 2024-04-26T20:11:50.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f266be00640 1 -- 172.21.15.94:0/329770033 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2678005440 con 0x563e91e24ae0 2024-04-26T20:11:50.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f269439a740 10 client.15087 fetch_fsmap learned FSMap version 24 2024-04-26T20:11:50.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.168+0000 7f269439a740 1 -- 172.21.15.94:0/329770033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563e91ea1560 con 0x563e91e24ae0 2024-04-26T20:11:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.169+0000 7f266be00640 1 -- 172.21.15.94:0/329770033 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2678005070 con 0x563e91e24ae0 2024-04-26T20:11:50.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.169+0000 7f266be00640 1 -- 172.21.15.94:0/329770033 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f267800fb70 con 0x563e91e24ae0 2024-04-26T20:11:50.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.169+0000 7f269439a740 10 client.15087 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:11:50.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.169+0000 7f269439a740 10 client.15087 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:11:50.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.169+0000 7f269439a740 20 client.15087 populate_metadata read hostname 'smithi094' 2024-04-26T20:11:50.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.169+0000 7f269439a740 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x563e91ea5e20 0x563e91ea4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:50.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.169+0000 7f269439a740 4 client.15087 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:11:50.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.169+0000 7f269439a740 1 -- 172.21.15.94:0/329770033 --> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] -- command(tid 0: {"prefix": "status"}) -- 0x563e91d90080 con 0x563e91ea5e20 2024-04-26T20:11:50.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.169+0000 7f268b400640 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x563e91ea5e20 0x563e91ea4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:50.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f268b400640 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x563e91ea5e20 0x563e91ea4b30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f201c89cde32f0f server_cookie=97a9b725afc9ca19 in_seq=0 out_seq=0 2024-04-26T20:11:50.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f266be00640 10 client.15087 ms_handle_connect on v2:172.21.15.103:6838/1048742250 2024-04-26T20:11:50.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f266be00640 1 -- 172.21.15.94:0/329770033 <== mds.0 v2:172.21.15.103:6838/1048742250 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x563e91d90080 con 0x563e91ea5e20 2024-04-26T20:11:50.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f266be00640 10 client.15087 handle_command_reply: tid=0 2024-04-26T20:11:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f269439a740 1 client.15087 shutdown 2024-04-26T20:11:50.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f269439a740 1 -- 172.21.15.94:0/329770033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e24ae0 msgr2=0x563e91e31a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:50.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f269439a740 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e24ae0 0x563e91e31a90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2678012420 tx=0x7f2678004fa0 comp rx=0 tx=0).stop 2024-04-26T20:11:50.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f269439a740 1 -- 172.21.15.94:0/329770033 shutdown_connections 2024-04-26T20:11:50.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f269439a740 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x563e91ea5e20 0x563e91ea4b30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:50.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f269439a740 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563e91e29c50 0x563e91e34ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:50.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f269439a740 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x563e91e24ae0 0x563e91e31a90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:50.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f269439a740 1 --2- 172.21.15.94:0/329770033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563e91e240f0 0x563e91e31550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:50.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f269439a740 1 -- 172.21.15.94:0/329770033 >> 172.21.15.94:0/329770033 conn(0x563e91db8d10 msgr2=0x563e91e2df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:50.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.170+0000 7f266be00640 0 client.15087 ms_handle_reset on v2:172.21.15.103:6838/1048742250 2024-04-26T20:11:50.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f269439a740 1 -- 172.21.15.94:0/329770033 shutdown_connections 2024-04-26T20:11:50.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f269439a740 1 -- 172.21.15.94:0/329770033 wait complete. 2024-04-26T20:11:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f269439a740 20 client.15087 trim_cache size 0 max 16384 2024-04-26T20:11:50.753 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-26T20:11:50.803 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T20:11:50.810 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-04-26T20:11:50.819 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24575, 2024-04-26T20:11:50.873 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-26T20:11:50.880 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-26T20:11:50.932 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-26T20:11:50.982 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 208.51297588, 2024-04-26T20:11:50.990 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 24, 2024-04-26T20:11:51.002 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 96, 2024-04-26T20:11:51.010 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 94, 2024-04-26T20:11:51.017 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 225.43947847699999 2024-04-26T20:11:51.025 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-26T20:11:51.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f2691600640 1 -- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26700a3c40 msgr2=0x7f26700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:51.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f2691600640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26700a3c40 0x7f26700a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x563e91e24ee0 tx=0x7f267c0d47d0 comp rx=0 tx=0).stop 2024-04-26T20:11:51.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f2691600640 1 -- 172.21.15.94:0/4162567648 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f268c102970 msgr2=0x7f268c19ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:51.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f2691600640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f268c102970 0x7f268c19ad50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e9c0 tx=0x7f267c00ee80 comp rx=0 tx=0).stop 2024-04-26T20:11:51.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f2691600640 1 -- 172.21.15.94:0/4162567648 shutdown_connections 2024-04-26T20:11:51.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f2691600640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26700a3c40 0x7f26700a6100 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:51.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f2691600640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f268c104520 0x7f268c1a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:51.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f2691600640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f268c103b60 0x7f268c19b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:51.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.171+0000 7f2691600640 1 --2- 172.21.15.94:0/4162567648 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f268c102970 0x7f268c19ad50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:51.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.172+0000 7f2691600640 1 -- 172.21.15.94:0/4162567648 >> 172.21.15.94:0/4162567648 conn(0x7f268c0fdee0 msgr2=0x7f268c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:51.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.172+0000 7f2691600640 1 -- 172.21.15.94:0/4162567648 shutdown_connections 2024-04-26T20:11:51.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:47.172+0000 7f2691600640 1 -- 172.21.15.94:0/4162567648 wait complete. 2024-04-26T20:11:51.324 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T20:11:51.331 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T20:11:51.349 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-26T20:11:51.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T20:11:51.361 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:48 smithi094 ceph-mon[28050]: pgmap v434: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:51.366 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:50 smithi094 ceph-mon[28050]: pgmap v435: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:51.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:48 smithi204 ceph-mon[32618]: pgmap v434: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:51.421 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:50 smithi204 ceph-mon[32618]: pgmap v435: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:51.459 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:48 smithi103 ceph-mon[32635]: pgmap v434: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:51.464 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:50 smithi103 ceph-mon[32635]: pgmap v435: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:11:51.921 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:11:52.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:52 smithi094 ceph-mon[28050]: pgmap v436: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:53.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:52 smithi103 ceph-mon[32635]: pgmap v436: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:53.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:52 smithi204 ceph-mon[32618]: pgmap v436: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:11:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.412+0000 7f1b3c400700 1 -- 172.21.15.94:0/2372786426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34103490 msgr2=0x7f1b341038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:54.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.412+0000 7f1b3c400700 1 --2- 172.21.15.94:0/2372786426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34103490 0x7f1b341038b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1b280099c0 tx=0x7f1b28009cd0 comp rx=0 tx=0).stop 2024-04-26T20:11:54.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.413+0000 7f1b3c400700 1 -- 172.21.15.94:0/2372786426 shutdown_connections 2024-04-26T20:11:54.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.413+0000 7f1b3c400700 1 --2- 172.21.15.94:0/2372786426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b34105030 0x7f1b3410b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:54.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.413+0000 7f1b3c400700 1 --2- 172.21.15.94:0/2372786426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34103490 0x7f1b341038b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:54.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.413+0000 7f1b3c400700 1 --2- 172.21.15.94:0/2372786426 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34104690 0x7f1b34104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:54.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.413+0000 7f1b3c400700 1 -- 172.21.15.94:0/2372786426 >> 172.21.15.94:0/2372786426 conn(0x7f1b340fe9f0 msgr2=0x7f1b34100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.413+0000 7f1b3c400700 1 -- 172.21.15.94:0/2372786426 shutdown_connections 2024-04-26T20:11:54.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.413+0000 7f1b3c400700 1 -- 172.21.15.94:0/2372786426 wait complete. 2024-04-26T20:11:54.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.414+0000 7f1b3c400700 1 Processor -- start 2024-04-26T20:11:54.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.414+0000 7f1b3c400700 1 -- start start 2024-04-26T20:11:54.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b3c400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b34103490 0x7f1b3419b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:54.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b3c400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34104690 0x7f1b3419be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:54.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b3c400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34105030 0x7f1b341a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:54.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b3c400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3410ef70 con 0x7f1b34103490 2024-04-26T20:11:54.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b3c400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3410edf0 con 0x7f1b34105030 2024-04-26T20:11:54.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b3c400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3410f0f0 con 0x7f1b34104690 2024-04-26T20:11:54.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b39000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34104690 0x7f1b3419be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:54.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b3a400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34105030 0x7f1b341a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b39a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b34103490 0x7f1b3419b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b3a400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34105030 0x7f1b341a2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:49166/0 (socket says 172.21.15.94:49166) 2024-04-26T20:11:54.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.415+0000 7f1b3a400700 1 -- 172.21.15.94:0/776697911 learned_addr learned my addr 172.21.15.94:0/776697911 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:54.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b3a400700 1 -- 172.21.15.94:0/776697911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34104690 msgr2=0x7f1b3419be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:54.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b3a400700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34104690 0x7f1b3419be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:54.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b3a400700 1 -- 172.21.15.94:0/776697911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b34103490 msgr2=0x7f1b3419b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:54.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b3a400700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b34103490 0x7f1b3419b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:54.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b3a400700 1 -- 172.21.15.94:0/776697911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b341a34a0 con 0x7f1b34105030 2024-04-26T20:11:54.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b39000700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34104690 0x7f1b3419be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:11:54.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b3a400700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34105030 0x7f1b341a2e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00ef30 tx=0x7f1b2c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:54.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b32000700 1 -- 172.21.15.94:0/776697911 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0092e0 con 0x7f1b34105030 2024-04-26T20:11:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b32000700 1 -- 172.21.15.94:0/776697911 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b2c004bb0 con 0x7f1b34105030 2024-04-26T20:11:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.416+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b341a3730 con 0x7f1b34105030 2024-04-26T20:11:54.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.417+0000 7f1b32000700 1 -- 172.21.15.94:0/776697911 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c004020 con 0x7f1b34105030 2024-04-26T20:11:54.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.417+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b341a3c70 con 0x7f1b34105030 2024-04-26T20:11:55.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.418+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b3406a110 con 0x7f1b34105030 2024-04-26T20:11:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.420+0000 7f1b32000700 1 -- 172.21.15.94:0/776697911 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1b2c01b030 con 0x7f1b34105030 2024-04-26T20:11:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.421+0000 7f1b32000700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1b1c0a3d50 0x7f1b1c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:55.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.421+0000 7f1b32000700 1 -- 172.21.15.94:0/776697911 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f1b2c0d03f0 con 0x7f1b34105030 2024-04-26T20:11:55.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.421+0000 7f1b39a00700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1b1c0a3d50 0x7f1b1c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:55.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.422+0000 7f1b39a00700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1b1c0a3d50 0x7f1b1c0a6210 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800b580 tx=0x7f1b28005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:55.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.425+0000 7f1b32000700 1 -- 172.21.15.94:0/776697911 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1b2c022020 con 0x7f1b34105030 2024-04-26T20:11:55.129 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:54 smithi103 ceph-mon[32635]: pgmap v437: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:55.136 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:11:55.142 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.148 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.180 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.187 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:54 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.701+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f1b34076d70 con 0x7f1b1c0a3d50 2024-04-26T20:11:55.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.703+0000 7f1b32000700 1 -- 172.21.15.94:0/776697911 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f1b34076d70 con 0x7f1b1c0a3d50 2024-04-26T20:11:55.212 INFO:teuthology.orchestra.run.smithi094.stdout:Backend: cephadm 2024-04-26T20:11:55.242 INFO:teuthology.orchestra.run.smithi094.stdout:Available: Yes 2024-04-26T20:11:55.275 INFO:teuthology.orchestra.run.smithi094.stdout:Paused: No 2024-04-26T20:11:55.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.706+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1b1c0a3d50 msgr2=0x7f1b1c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:55.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.706+0000 7f1b3c400700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1b1c0a3d50 0x7f1b1c0a6210 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800b580 tx=0x7f1b28005fb0 comp rx=0 tx=0).stop 2024-04-26T20:11:55.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34105030 msgr2=0x7f1b341a2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34105030 0x7f1b341a2e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00ef30 tx=0x7f1b2c00c3a0 comp rx=0 tx=0).stop 2024-04-26T20:11:55.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 shutdown_connections 2024-04-26T20:11:55.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1b1c0a3d50 0x7f1b1c0a6210 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:55.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b34105030 0x7f1b341a2e00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:55.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b34103490 0x7f1b3419b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:55.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 --2- 172.21.15.94:0/776697911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34104690 0x7f1b3419be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:55.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 >> 172.21.15.94:0/776697911 conn(0x7f1b340fe9f0 msgr2=0x7f1b34075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:55.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 shutdown_connections 2024-04-26T20:11:55.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:54.707+0000 7f1b3c400700 1 -- 172.21.15.94:0/776697911 wait complete. 2024-04-26T20:11:55.439 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:54 smithi094 ceph-mon[28050]: pgmap v437: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:55.445 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:11:55.453 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.459 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.483 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.538 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:54 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.550 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:54 smithi204 ceph-mon[32618]: pgmap v437: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:11:55.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:11:55.568 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.585 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.593 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:54 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:55.601 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T20:11:56.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:55 smithi103 ceph-mon[32635]: from='client.25196 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:11:56.040 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:11:56.089 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:11:56.095 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:56.101 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:11:56.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:55 smithi204 ceph-mon[32618]: from='client.25196 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:11:56.137 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:11:56.194 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:11:56.198 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:56.248 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:55 smithi094 ceph-mon[28050]: from='client.25196 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:11:56.280 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:11:56.285 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:11:56.290 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:11:56.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:56 smithi094 ceph-mon[28050]: pgmap v438: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:57.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:56 smithi103 ceph-mon[32635]: pgmap v438: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:57.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:56 smithi204 ceph-mon[32618]: pgmap v438: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:11:58.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.266+0000 7fde77000700 1 -- 172.21.15.94:0/765999780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde7010b2a0 msgr2=0x7fde7010d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.266+0000 7fde77000700 1 --2- 172.21.15.94:0/765999780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde7010b2a0 0x7fde7010d730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fde6800b0a0 tx=0x7fde6800b3b0 comp rx=0 tx=0).stop 2024-04-26T20:11:58.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.266+0000 7fde77000700 1 -- 172.21.15.94:0/765999780 shutdown_connections 2024-04-26T20:11:58.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.266+0000 7fde77000700 1 --2- 172.21.15.94:0/765999780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde7010b2a0 0x7fde7010d730 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:58.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.266+0000 7fde77000700 1 --2- 172.21.15.94:0/765999780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde700773f0 0x7fde70075850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:58.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.266+0000 7fde77000700 1 --2- 172.21.15.94:0/765999780 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde70075e20 0x7fde700762a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:58.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.266+0000 7fde77000700 1 -- 172.21.15.94:0/765999780 >> 172.21.15.94:0/765999780 conn(0x7fde700fe960 msgr2=0x7fde70100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:58.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.266+0000 7fde77000700 1 -- 172.21.15.94:0/765999780 shutdown_connections 2024-04-26T20:11:58.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.266+0000 7fde77000700 1 -- 172.21.15.94:0/765999780 wait complete. 2024-04-26T20:11:58.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.268+0000 7fde77000700 1 Processor -- start 2024-04-26T20:11:58.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.268+0000 7fde77000700 1 -- start start 2024-04-26T20:11:58.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde77000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde70075e20 0x7fde7019b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:58.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde77000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700773f0 0x7fde7019bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:58.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde77000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde7010b2a0 0x7fde701a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:58.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde77000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fde70111090 con 0x7fde7010b2a0 2024-04-26T20:11:58.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde77000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde70110f10 con 0x7fde700773f0 2024-04-26T20:11:58.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde77000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fde70111210 con 0x7fde70075e20 2024-04-26T20:11:58.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde76600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde7010b2a0 0x7fde701a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:58.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde75c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde70075e20 0x7fde7019b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:58.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde75200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700773f0 0x7fde7019bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:58.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde75c00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde70075e20 0x7fde7019b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:60534/0 (socket says 172.21.15.94:60534) 2024-04-26T20:11:58.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde75c00700 1 -- 172.21.15.94:0/647270059 learned_addr learned my addr 172.21.15.94:0/647270059 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:11:58.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde76600700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde7010b2a0 0x7fde701a2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51880/0 (socket says 172.21.15.94:51880) 2024-04-26T20:11:58.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.269+0000 7fde75200700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700773f0 0x7fde7019bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:49184/0 (socket says 172.21.15.94:49184) 2024-04-26T20:11:58.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde76600700 1 -- 172.21.15.94:0/647270059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde70075e20 msgr2=0x7fde7019b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:58.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde76600700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde70075e20 0x7fde7019b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:58.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde76600700 1 -- 172.21.15.94:0/647270059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700773f0 msgr2=0x7fde7019bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:58.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde76600700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700773f0 0x7fde7019bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:58.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde76600700 1 -- 172.21.15.94:0/647270059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde701a32e0 con 0x7fde7010b2a0 2024-04-26T20:11:58.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde76600700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde7010b2a0 0x7fde701a2c40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fde68015000 tx=0x7fde68004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:58.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde6e000700 1 -- 172.21.15.94:0/647270059 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde68021070 con 0x7fde7010b2a0 2024-04-26T20:11:58.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde6e000700 1 -- 172.21.15.94:0/647270059 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde68007b80 con 0x7fde7010b2a0 2024-04-26T20:11:58.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde6e000700 1 -- 172.21.15.94:0/647270059 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde680052a0 con 0x7fde7010b2a0 2024-04-26T20:11:58.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.270+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde701a3570 con 0x7fde7010b2a0 2024-04-26T20:11:58.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.271+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde701a3ab0 con 0x7fde7010b2a0 2024-04-26T20:11:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.272+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde7006a110 con 0x7fde7010b2a0 2024-04-26T20:11:58.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.275+0000 7fde6e000700 1 -- 172.21.15.94:0/647270059 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fde68024070 con 0x7fde7010b2a0 2024-04-26T20:11:59.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.277+0000 7fde6e000700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fde580a3cd0 0x7fde580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:11:59.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.277+0000 7fde6e000700 1 -- 172.21.15.94:0/647270059 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fde6800f020 con 0x7fde7010b2a0 2024-04-26T20:11:59.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.277+0000 7fde75c00700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fde580a3cd0 0x7fde580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:11:59.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.278+0000 7fde75c00700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fde580a3cd0 0x7fde580a6190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fde64009d20 tx=0x7fde64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:11:59.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.279+0000 7fde6e000700 1 -- 172.21.15.94:0/647270059 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fde6802b020 con 0x7fde7010b2a0 2024-04-26T20:11:59.097 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:58 smithi103 ceph-mon[32635]: pgmap v439: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:59.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.517+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fde70063860 con 0x7fde580a3cd0 2024-04-26T20:11:59.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.541+0000 7fde6e000700 1 -- 172.21.15.94:0/647270059 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fde70063860 con 0x7fde580a3cd0 2024-04-26T20:11:59.175 INFO:teuthology.orchestra.run.smithi094.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T20:11:59.181 INFO:teuthology.orchestra.run.smithi094.stdout:mds.a smithi094 running (4m) 66s ago 3m 17.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 9c1d2dbf8e40 2024-04-26T20:11:59.188 INFO:teuthology.orchestra.run.smithi094.stdout:mds.b smithi103 running (3m) 104s ago 3m 25.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 f37d75c0bce3 2024-04-26T20:11:59.195 INFO:teuthology.orchestra.run.smithi094.stdout:mds.c smithi204 running (4m) 104s ago 4m 16.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 941a731dbf39 2024-04-26T20:11:59.203 INFO:teuthology.orchestra.run.smithi094.stdout:mds.d smithi094 running (4m) 66s ago 4m 13.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 394cd22bae4c 2024-04-26T20:11:59.211 INFO:teuthology.orchestra.run.smithi094.stdout:mds.e smithi103 running (4m) 104s ago 4m 13.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 e40f6bcc56a9 2024-04-26T20:11:59.220 INFO:teuthology.orchestra.run.smithi094.stdout:mds.f smithi204 running (4m) 104s ago 4m 16.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 29b9dc616261 2024-04-26T20:11:59.265 INFO:teuthology.orchestra.run.smithi094.stdout:mds.g smithi094 running (4m) 66s ago 4m 12.9M - 19.0.0-3306-gbc612ffb 55dc54019ad8 6bc36881c32d 2024-04-26T20:11:59.270 INFO:teuthology.orchestra.run.smithi094.stdout:mds.h smithi103 running (4m) 104s ago 4m 21.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 11411d6eab46 2024-04-26T20:11:59.309 INFO:teuthology.orchestra.run.smithi094.stdout:mds.i smithi204 running (4m) 104s ago 4m 17.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 ba280396131d 2024-04-26T20:11:59.340 INFO:teuthology.orchestra.run.smithi094.stdout:mds.j smithi094 running (4m) 66s ago 4m 13.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 c8c81c34b8fb 2024-04-26T20:11:59.346 INFO:teuthology.orchestra.run.smithi094.stdout:mds.k smithi103 running (4m) 104s ago 4m 13.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 93948c78258d 2024-04-26T20:11:59.352 INFO:teuthology.orchestra.run.smithi094.stdout:mds.l smithi204 running (4m) 104s ago 4m 13.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 ccaadfeb5f0e 2024-04-26T20:11:59.359 INFO:teuthology.orchestra.run.smithi094.stdout:mgr.x smithi094 *:9283,8765 running (12m) 66s ago 12m 517M - 19.0.0-3306-gbc612ffb 55dc54019ad8 dd6491447774 2024-04-26T20:11:59.381 INFO:teuthology.orchestra.run.smithi094.stdout:mgr.y smithi103 *:8443,8765 running (10m) 104s ago 10m 454M - 19.0.0-3306-gbc612ffb 55dc54019ad8 d8d849286148 2024-04-26T20:11:59.387 INFO:teuthology.orchestra.run.smithi094.stdout:mgr.z smithi204 *:8443,8765 running (10m) 104s ago 10m 453M - 19.0.0-3306-gbc612ffb 55dc54019ad8 aa57009e2a3c 2024-04-26T20:11:59.438 INFO:teuthology.orchestra.run.smithi094.stdout:mon.a smithi094 running (12m) 66s ago 13m 49.7M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 276d9903869a 2024-04-26T20:11:59.444 INFO:teuthology.orchestra.run.smithi094.stdout:mon.b smithi103 running (10m) 104s ago 10m 42.8M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 e6e36d2770df 2024-04-26T20:11:59.449 INFO:teuthology.orchestra.run.smithi094.stdout:mon.c smithi204 running (10m) 104s ago 10m 40.9M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 f5675512b043 2024-04-26T20:11:59.457 INFO:teuthology.orchestra.run.smithi094.stdout:osd.0 smithi094 running (10m) 66s ago 10m 52.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 09ebc3c5b363 2024-04-26T20:11:59.463 INFO:teuthology.orchestra.run.smithi094.stdout:osd.1 smithi103 running (9m) 104s ago 9m 56.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 664e5e84dd13 2024-04-26T20:11:59.468 INFO:teuthology.orchestra.run.smithi094.stdout:osd.2 smithi204 running (9m) 104s ago 9m 46.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 dbf968be4d73 2024-04-26T20:11:59.473 INFO:teuthology.orchestra.run.smithi094.stdout:osd.3 smithi094 running (8m) 66s ago 8m 43.9M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 3692f4ec0463 2024-04-26T20:11:59.479 INFO:teuthology.orchestra.run.smithi094.stdout:osd.4 smithi103 running (8m) 104s ago 8m 43.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 06578b1ee824 2024-04-26T20:11:59.528 INFO:teuthology.orchestra.run.smithi094.stdout:osd.5 smithi204 running (7m) 104s ago 7m 47.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 6e8b5837f105 2024-04-26T20:11:59.535 INFO:teuthology.orchestra.run.smithi094.stdout:osd.6 smithi094 running (7m) 66s ago 7m 48.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 e3e4ffa0eb63 2024-04-26T20:11:59.582 INFO:teuthology.orchestra.run.smithi094.stdout:osd.7 smithi103 running (6m) 104s ago 6m 53.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 07883f983056 2024-04-26T20:11:59.588 INFO:teuthology.orchestra.run.smithi094.stdout:osd.8 smithi204 running (6m) 104s ago 6m 54.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 226eaa96b768 2024-04-26T20:11:59.595 INFO:teuthology.orchestra.run.smithi094.stdout:osd.9 smithi094 running (5m) 66s ago 5m 49.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 9f24579bc988 2024-04-26T20:11:59.602 INFO:teuthology.orchestra.run.smithi094.stdout:osd.10 smithi103 running (5m) 104s ago 5m 47.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 b51b62f432c3 2024-04-26T20:11:59.609 INFO:teuthology.orchestra.run.smithi094.stdout:osd.11 smithi204 running (4m) 104s ago 4m 52.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 8eae1083e651 2024-04-26T20:11:59.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fde580a3cd0 msgr2=0x7fde580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:59.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fde580a3cd0 0x7fde580a6190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fde64009d20 tx=0x7fde64009260 comp rx=0 tx=0).stop 2024-04-26T20:11:59.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde7010b2a0 msgr2=0x7fde701a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:11:59.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde7010b2a0 0x7fde701a2c40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fde68015000 tx=0x7fde68004c60 comp rx=0 tx=0).stop 2024-04-26T20:11:59.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 shutdown_connections 2024-04-26T20:11:59.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde7010b2a0 0x7fde701a2c40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:59.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde70075e20 0x7fde7019b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:59.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fde580a3cd0 0x7fde580a6190 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:59.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 --2- 172.21.15.94:0/647270059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700773f0 0x7fde7019bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:11:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 >> 172.21.15.94:0/647270059 conn(0x7fde700fe960 msgr2=0x7fde701071f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:11:59.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 shutdown_connections 2024-04-26T20:11:59.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:11:58.544+0000 7fde77000700 1 -- 172.21.15.94:0/647270059 wait complete. 2024-04-26T20:11:59.905 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:58 smithi094 ceph-mon[28050]: pgmap v439: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:11:59.911 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T20:11:59.920 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:58 smithi204 ceph-mon[32618]: pgmap v439: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:00.012 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:11:59 smithi094 ceph-mon[28050]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:00.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:11:59 smithi103 ceph-mon[32635]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:00.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:11:59 smithi204 ceph-mon[32618]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:00.388 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:12:00.875 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:00 smithi094 ceph-mon[28050]: pgmap v440: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:12:01.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:00 smithi103 ceph-mon[32635]: pgmap v440: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:12:01.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:00 smithi204 ceph-mon[32618]: pgmap v440: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:12:02.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:01 smithi103 ceph-mon[32635]: pgmap v441: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:02.058 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:01 smithi094 ceph-mon[28050]: pgmap v441: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:02.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:01 smithi204 ceph-mon[32618]: pgmap v441: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:02.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.573+0000 7fe22d000700 1 -- 172.21.15.94:0/3908944791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 msgr2=0x7fe22810da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:02.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.573+0000 7fe22d000700 1 --2- 172.21.15.94:0/3908944791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 0x7fe22810da10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe2200099c0 tx=0x7fe220009cd0 comp rx=0 tx=0).stop 2024-04-26T20:12:02.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.573+0000 7fe22d000700 1 -- 172.21.15.94:0/3908944791 shutdown_connections 2024-04-26T20:12:02.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.573+0000 7fe22d000700 1 --2- 172.21.15.94:0/3908944791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 0x7fe22810da10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:02.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.573+0000 7fe22d000700 1 --2- 172.21.15.94:0/3908944791 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2281055c0 0x7fe2281059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.573+0000 7fe22d000700 1 --2- 172.21.15.94:0/3908944791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2281067c0 0x7fe228106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:02.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.573+0000 7fe22d000700 1 -- 172.21.15.94:0/3908944791 >> 172.21.15.94:0/3908944791 conn(0x7fe228100b40 msgr2=0x7fe228102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:02.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.573+0000 7fe22d000700 1 -- 172.21.15.94:0/3908944791 shutdown_connections 2024-04-26T20:12:02.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.573+0000 7fe22d000700 1 -- 172.21.15.94:0/3908944791 wait complete. 2024-04-26T20:12:02.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.574+0000 7fe22d000700 1 Processor -- start 2024-04-26T20:12:02.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe22d000700 1 -- start start 2024-04-26T20:12:02.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe22d000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2281055c0 0x7fe22819b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:02.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe22d000700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2281067c0 0x7fe22819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:02.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe22d000700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 0x7fe2281a2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe22d000700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe228111030 con 0x7fe228107160 2024-04-26T20:12:03.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe22d000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe228110eb0 con 0x7fe2281055c0 2024-04-26T20:12:03.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe22d000700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe2281111b0 con 0x7fe2281067c0 2024-04-26T20:12:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe227e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 0x7fe2281a2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:03.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe227400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2281055c0 0x7fe22819b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe227400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2281055c0 0x7fe22819b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:53894/0 (socket says 172.21.15.94:53894) 2024-04-26T20:12:03.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe227e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 0x7fe2281a2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57904/0 (socket says 172.21.15.94:57904) 2024-04-26T20:12:03.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.575+0000 7fe227400700 1 -- 172.21.15.94:0/2103107732 learned_addr learned my addr 172.21.15.94:0/2103107732 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:03.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.576+0000 7fe21ea00700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2281067c0 0x7fe22819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:03.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.576+0000 7fe227e00700 1 -- 172.21.15.94:0/2103107732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2281067c0 msgr2=0x7fe22819bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.576+0000 7fe227e00700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2281067c0 0x7fe22819bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:03.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.576+0000 7fe227e00700 1 -- 172.21.15.94:0/2103107732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2281055c0 msgr2=0x7fe22819b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:03.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.576+0000 7fe227e00700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2281055c0 0x7fe22819b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:03.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.576+0000 7fe227e00700 1 -- 172.21.15.94:0/2103107732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2281a31f0 con 0x7fe228107160 2024-04-26T20:12:03.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.576+0000 7fe227e00700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 0x7fe2281a2b50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe2200036f0 tx=0x7fe220003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:03.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.577+0000 7fe224c00700 1 -- 172.21.15.94:0/2103107732 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2200050c0 con 0x7fe228107160 2024-04-26T20:12:03.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.577+0000 7fe224c00700 1 -- 172.21.15.94:0/2103107732 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe22000f460 con 0x7fe228107160 2024-04-26T20:12:03.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.577+0000 7fe224c00700 1 -- 172.21.15.94:0/2103107732 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220017630 con 0x7fe228107160 2024-04-26T20:12:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.577+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2281a3480 con 0x7fe228107160 2024-04-26T20:12:03.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.577+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2281a3880 con 0x7fe228107160 2024-04-26T20:12:03.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.578+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe22806a110 con 0x7fe228107160 2024-04-26T20:12:03.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.580+0000 7fe224c00700 1 -- 172.21.15.94:0/2103107732 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe22000f610 con 0x7fe228107160 2024-04-26T20:12:03.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.581+0000 7fe224c00700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe20c0a3d50 0x7fe20c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:03.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.582+0000 7fe224c00700 1 -- 172.21.15.94:0/2103107732 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fe2200d1600 con 0x7fe228107160 2024-04-26T20:12:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.582+0000 7fe227400700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe20c0a3d50 0x7fe20c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:03.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.582+0000 7fe227400700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe20c0a3d50 0x7fe20c0a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe21800f5e0 tx=0x7fe21800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:03.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.586+0000 7fe224c00700 1 -- 172.21.15.94:0/2103107732 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe220013070 con 0x7fe228107160 2024-04-26T20:12:03.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.822+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fe228063860 con 0x7fe20c0a3d50 2024-04-26T20:12:03.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.836+0000 7fe224c00700 1 -- 172.21.15.94:0/2103107732 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fe228063860 con 0x7fe20c0a3d50 2024-04-26T20:12:03.534 INFO:teuthology.orchestra.run.smithi094.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T20:12:03.547 INFO:teuthology.orchestra.run.smithi094.stdout:mds.all 12/12 108s ago 4m smithi094=a;smithi094=d;smithi094=g;smithi094=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-04-26T20:12:03.606 INFO:teuthology.orchestra.run.smithi094.stdout:mgr 3/3 108s ago 10m smithi094=x;smithi103=y;smithi204=z;count:3 2024-04-26T20:12:03.651 INFO:teuthology.orchestra.run.smithi094.stdout:mon 3/3 108s ago 11m smithi094:172.21.15.94=a;smithi103:172.21.15.103=b;smithi204:172.21.15.204=c;count:3 2024-04-26T20:12:03.660 INFO:teuthology.orchestra.run.smithi094.stdout:osd 12 108s ago - 2024-04-26T20:12:03.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe20c0a3d50 msgr2=0x7fe20c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:03.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe20c0a3d50 0x7fe20c0a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe21800f5e0 tx=0x7fe21800b040 comp rx=0 tx=0).stop 2024-04-26T20:12:03.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 msgr2=0x7fe2281a2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:03.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 0x7fe2281a2b50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe2200036f0 tx=0x7fe220003af0 comp rx=0 tx=0).stop 2024-04-26T20:12:03.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 shutdown_connections 2024-04-26T20:12:03.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe228107160 0x7fe2281a2b50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:03.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe20c0a3d50 0x7fe20c0a6210 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:03.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2281055c0 0x7fe22819b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:03.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 --2- 172.21.15.94:0/2103107732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2281067c0 0x7fe22819bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:03.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.839+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 >> 172.21.15.94:0/2103107732 conn(0x7fe228100b40 msgr2=0x7fe2281099f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:03.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.840+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 shutdown_connections 2024-04-26T20:12:03.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:02.840+0000 7fe22d000700 1 -- 172.21.15.94:0/2103107732 wait complete. 2024-04-26T20:12:03.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T20:12:04.326 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:12:04.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:04 smithi094 ceph-mon[28050]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:04.749 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:04 smithi094 ceph-mon[28050]: pgmap v442: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:05.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:04 smithi103 ceph-mon[32635]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:05.044 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:04 smithi103 ceph-mon[32635]: pgmap v442: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:05.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:04 smithi204 ceph-mon[32618]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:05.140 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:04 smithi204 ceph-mon[32618]: pgmap v442: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:06.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.571+0000 7ff5b3400700 1 -- 172.21.15.94:0/2388485137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5ac1056c0 msgr2=0x7ff5ac105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:06.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.572+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2388485137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5ac1056c0 0x7ff5ac105ae0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00099c0 tx=0x7ff5a0009cd0 comp rx=0 tx=0).stop 2024-04-26T20:12:06.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.572+0000 7ff5b3400700 1 -- 172.21.15.94:0/2388485137 shutdown_connections 2024-04-26T20:12:06.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.572+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2388485137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5ac107260 0x7ff5ac10db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:06.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.572+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2388485137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5ac1056c0 0x7ff5ac105ae0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.572+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2388485137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac1068c0 0x7ff5ac106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:06.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.572+0000 7ff5b3400700 1 -- 172.21.15.94:0/2388485137 >> 172.21.15.94:0/2388485137 conn(0x7ff5ac100c80 msgr2=0x7ff5ac1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:06.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.572+0000 7ff5b3400700 1 -- 172.21.15.94:0/2388485137 shutdown_connections 2024-04-26T20:12:06.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.572+0000 7ff5b3400700 1 -- 172.21.15.94:0/2388485137 wait complete. 2024-04-26T20:12:06.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.573+0000 7ff5b3400700 1 Processor -- start 2024-04-26T20:12:06.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.573+0000 7ff5b3400700 1 -- start start 2024-04-26T20:12:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b3400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5ac1056c0 0x7ff5ac19b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:06.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b3400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5ac1068c0 0x7ff5ac19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:06.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b3400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac107260 0x7ff5ac1a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:06.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b3400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ac1111a0 con 0x7ff5ac1056c0 2024-04-26T20:12:06.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b3400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ac111020 con 0x7ff5ac1068c0 2024-04-26T20:12:06.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b3400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ac111320 con 0x7ff5ac107260 2024-04-26T20:12:06.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b0a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5ac1056c0 0x7ff5ac19b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:07.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b0a00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5ac1056c0 0x7ff5ac19b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57928/0 (socket says 172.21.15.94:57928) 2024-04-26T20:12:07.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b0a00700 1 -- 172.21.15.94:0/2077263393 learned_addr learned my addr 172.21.15.94:0/2077263393 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:07.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5abe00700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5ac1068c0 0x7ff5ac19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:07.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.574+0000 7ff5b1400700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac107260 0x7ff5ac1a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:07.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5abe00700 1 -- 172.21.15.94:0/2077263393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac107260 msgr2=0x7ff5ac1a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:07.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5abe00700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac107260 0x7ff5ac1a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:07.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5abe00700 1 -- 172.21.15.94:0/2077263393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5ac1056c0 msgr2=0x7ff5ac19b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:07.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5abe00700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5ac1056c0 0x7ff5ac19b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:07.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5abe00700 1 -- 172.21.15.94:0/2077263393 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ac1a3480 con 0x7ff5ac1068c0 2024-04-26T20:12:07.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5abe00700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5ac1068c0 0x7ff5ac19bd60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff59800e9f0 tx=0x7ff59800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:07.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5a9600700 1 -- 172.21.15.94:0/2077263393 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598004220 con 0x7ff5ac1068c0 2024-04-26T20:12:07.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5a9600700 1 -- 172.21.15.94:0/2077263393 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5980043c0 con 0x7ff5ac1068c0 2024-04-26T20:12:07.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ac1a3770 con 0x7ff5ac1068c0 2024-04-26T20:12:07.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.575+0000 7ff5a9600700 1 -- 172.21.15.94:0/2077263393 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59800ce60 con 0x7ff5ac1068c0 2024-04-26T20:12:07.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.576+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5ac1a3cb0 con 0x7ff5ac1068c0 2024-04-26T20:12:07.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.577+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5ac06a110 con 0x7ff5ac1068c0 2024-04-26T20:12:07.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.579+0000 7ff5a9600700 1 -- 172.21.15.94:0/2077263393 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff598005330 con 0x7ff5ac1068c0 2024-04-26T20:12:07.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.580+0000 7ff5a9600700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff5940a3d50 0x7ff5940a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:07.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.580+0000 7ff5a9600700 1 -- 172.21.15.94:0/2077263393 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7ff5980d0b80 con 0x7ff5ac1068c0 2024-04-26T20:12:07.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.580+0000 7ff5b0a00700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff5940a3d50 0x7ff5940a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:07.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.581+0000 7ff5b0a00700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff5940a3d50 0x7ff5940a6210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0003820 tx=0x7ff5a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:07.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.582+0000 7ff5a9600700 1 -- 172.21.15.94:0/2077263393 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff59809d060 con 0x7ff5ac1068c0 2024-04-26T20:12:07.401 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:06 smithi204 ceph-mon[32618]: pgmap v443: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:07.408 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:06 smithi103 ceph-mon[32635]: pgmap v443: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:07.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.816+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7ff5ac063860 con 0x7ff5940a3d50 2024-04-26T20:12:07.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.818+0000 7ff5a9600700 1 -- 172.21.15.94:0/2077263393 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7ff5ac063860 con 0x7ff5940a3d50 2024-04-26T20:12:07.478 INFO:teuthology.orchestra.run.smithi094.stdout:HOST ADDR LABELS STATUS 2024-04-26T20:12:07.531 INFO:teuthology.orchestra.run.smithi094.stdout:smithi094 172.21.15.94 2024-04-26T20:12:07.536 INFO:teuthology.orchestra.run.smithi094.stdout:smithi103 172.21.15.103 2024-04-26T20:12:07.542 INFO:teuthology.orchestra.run.smithi094.stdout:smithi204 172.21.15.204 2024-04-26T20:12:07.549 INFO:teuthology.orchestra.run.smithi094.stdout:3 hosts in cluster 2024-04-26T20:12:07.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.821+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff5940a3d50 msgr2=0x7ff5940a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:07.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.821+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff5940a3d50 0x7ff5940a6210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0003820 tx=0x7ff5a0005fb0 comp rx=0 tx=0).stop 2024-04-26T20:12:07.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.821+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5ac1068c0 msgr2=0x7ff5ac19bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:07.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.821+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5ac1068c0 0x7ff5ac19bd60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff59800e9f0 tx=0x7ff59800ed00 comp rx=0 tx=0).stop 2024-04-26T20:12:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.822+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 shutdown_connections 2024-04-26T20:12:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.822+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac107260 0x7ff5ac1a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:07.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.822+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5ac1056c0 0x7ff5ac19b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:07.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.822+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff5940a3d50 0x7ff5940a6210 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:07.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.822+0000 7ff5b3400700 1 --2- 172.21.15.94:0/2077263393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5ac1068c0 0x7ff5ac19bd60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:07.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.822+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 >> 172.21.15.94:0/2077263393 conn(0x7ff5ac100c80 msgr2=0x7ff5ac109c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:07.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.822+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 shutdown_connections 2024-04-26T20:12:07.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:06.822+0000 7ff5b3400700 1 -- 172.21.15.94:0/2077263393 wait complete. 2024-04-26T20:12:07.811 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:06 smithi094 ceph-mon[28050]: pgmap v443: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:07.875 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T20:12:08.325 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:12:08.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:08 smithi094 ceph-mon[28050]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:08.994 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:08 smithi094 ceph-mon[28050]: pgmap v444: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:09.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:08 smithi103 ceph-mon[32635]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:09.040 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:08 smithi103 ceph-mon[32635]: pgmap v444: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:09.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:08 smithi204 ceph-mon[32618]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:09.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:08 smithi204 ceph-mon[32618]: pgmap v444: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:10.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.684+0000 7fddb6800700 1 -- 172.21.15.94:0/2014056186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 msgr2=0x7fddb0106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:10.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.684+0000 7fddb6800700 1 --2- 172.21.15.94:0/2014056186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 0x7fddb0106c20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0098f0 tx=0x7fdd9c009c00 comp rx=0 tx=0).stop 2024-04-26T20:12:10.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.684+0000 7fddb6800700 1 -- 172.21.15.94:0/2014056186 shutdown_connections 2024-04-26T20:12:10.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.684+0000 7fddb6800700 1 --2- 172.21.15.94:0/2014056186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddb0107160 0x7fddb010da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.684+0000 7fddb6800700 1 --2- 172.21.15.94:0/2014056186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddb01055c0 0x7fddb01059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.684+0000 7fddb6800700 1 --2- 172.21.15.94:0/2014056186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 0x7fddb0106c20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.684+0000 7fddb6800700 1 -- 172.21.15.94:0/2014056186 >> 172.21.15.94:0/2014056186 conn(0x7fddb0100b60 msgr2=0x7fddb0102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:10.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.685+0000 7fddb6800700 1 -- 172.21.15.94:0/2014056186 shutdown_connections 2024-04-26T20:12:10.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.685+0000 7fddb6800700 1 -- 172.21.15.94:0/2014056186 wait complete. 2024-04-26T20:12:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.686+0000 7fddb6800700 1 Processor -- start 2024-04-26T20:12:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.686+0000 7fddb6800700 1 -- start start 2024-04-26T20:12:10.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.686+0000 7fddb6800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddb01055c0 0x7fddb019b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:10.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.686+0000 7fddb6800700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 0x7fddb019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:10.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.686+0000 7fddb6800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddb0107160 0x7fddb01a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.686+0000 7fddb6800700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fddb01110a0 con 0x7fddb0107160 2024-04-26T20:12:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.686+0000 7fddb6800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fddb0110f20 con 0x7fddb01055c0 2024-04-26T20:12:10.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.686+0000 7fddb6800700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fddb0111220 con 0x7fddb01067c0 2024-04-26T20:12:10.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddb01055c0 0x7fddb019b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:10.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb4a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 0x7fddb019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:11.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5e00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddb0107160 0x7fddb01a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:11.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddb01055c0 0x7fddb019b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:53922/0 (socket says 172.21.15.94:53922) 2024-04-26T20:12:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb4a00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 0x7fddb019bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:49296/0 (socket says 172.21.15.94:49296) 2024-04-26T20:12:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5400700 1 -- 172.21.15.94:0/63933234 learned_addr learned my addr 172.21.15.94:0/63933234 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:11.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5400700 1 -- 172.21.15.94:0/63933234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 msgr2=0x7fddb019bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5400700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 0x7fddb019bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:11.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5400700 1 -- 172.21.15.94:0/63933234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddb0107160 msgr2=0x7fddb01a2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:11.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5400700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddb0107160 0x7fddb01a2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:11.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5400700 1 -- 172.21.15.94:0/63933234 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb01a33b0 con 0x7fddb01055c0 2024-04-26T20:12:11.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb4a00700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 0x7fddb019bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:11.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.687+0000 7fddb5400700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddb01055c0 0x7fddb019b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdda400e9f0 tx=0x7fdda400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:11.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.688+0000 7fddae000700 1 -- 172.21.15.94:0/63933234 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4004220 con 0x7fddb01055c0 2024-04-26T20:12:11.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.688+0000 7fddae000700 1 -- 172.21.15.94:0/63933234 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdda40043c0 con 0x7fddb01055c0 2024-04-26T20:12:11.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.688+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb01a36a0 con 0x7fddb01055c0 2024-04-26T20:12:11.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.688+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb01a3c30 con 0x7fddb01055c0 2024-04-26T20:12:11.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.688+0000 7fddae000700 1 -- 172.21.15.94:0/63933234 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4003bc0 con 0x7fddb01055c0 2024-04-26T20:12:11.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.689+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd90005360 con 0x7fddb01055c0 2024-04-26T20:12:11.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.691+0000 7fddae000700 1 -- 172.21.15.94:0/63933234 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdda400c7c0 con 0x7fddb01055c0 2024-04-26T20:12:11.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.692+0000 7fddae000700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd980a3d50 0x7fdd980a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:11.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.692+0000 7fddae000700 1 -- 172.21.15.94:0/63933234 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fdda40d0a30 con 0x7fddb01055c0 2024-04-26T20:12:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.693+0000 7fddb4a00700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd980a3d50 0x7fdd980a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:11.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.693+0000 7fddb4a00700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd980a3d50 0x7fdd980a6210 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00fb20 tx=0x7fdd9c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:11.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.697+0000 7fddae000700 1 -- 172.21.15.94:0/63933234 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdda409ce80 con 0x7fddb01055c0 2024-04-26T20:12:11.494 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:10 smithi204 ceph-mon[32618]: pgmap v445: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:11.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:10 smithi103 ceph-mon[32635]: pgmap v445: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:11.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.932+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fdd90000bf0 con 0x7fdd980a3d50 2024-04-26T20:12:11.522 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:10 smithi094 ceph-mon[28050]: pgmap v445: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:11.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.935+0000 7fddae000700 1 -- 172.21.15.94:0/63933234 <== mgr.14152 v2:172.21.15.94:6800/2112565670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fdd90000bf0 con 0x7fdd980a3d50 2024-04-26T20:12:11.538 INFO:teuthology.orchestra.run.smithi094.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T20:12:11.545 INFO:teuthology.orchestra.run.smithi094.stdout:smithi094 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513001L480DGN 447G No 5m ago Has a FileSystem, LVM detected 2024-04-26T20:12:11.553 INFO:teuthology.orchestra.run.smithi094.stdout:smithi103 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204012Z400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-04-26T20:12:11.561 INFO:teuthology.orchestra.run.smithi094.stdout:smithi204 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GD400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-04-26T20:12:11.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd980a3d50 msgr2=0x7fdd980a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:11.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd980a3d50 0x7fdd980a6210 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00fb20 tx=0x7fdd9c019040 comp rx=0 tx=0).stop 2024-04-26T20:12:11.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddb01055c0 msgr2=0x7fddb019b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:11.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddb01055c0 0x7fddb019b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdda400e9f0 tx=0x7fdda400ed00 comp rx=0 tx=0).stop 2024-04-26T20:12:11.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 shutdown_connections 2024-04-26T20:12:11.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddb0107160 0x7fddb01a2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:11.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fdd980a3d50 0x7fdd980a6210 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddb01055c0 0x7fddb019b750 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:11.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 --2- 172.21.15.94:0/63933234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddb01067c0 0x7fddb019bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:11.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.939+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 >> 172.21.15.94:0/63933234 conn(0x7fddb0100b60 msgr2=0x7fddb0109b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:11.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.940+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 shutdown_connections 2024-04-26T20:12:11.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:10.940+0000 7fddb6800700 1 -- 172.21.15.94:0/63933234 wait complete. 2024-04-26T20:12:11.912 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T20:12:11.944 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-26T20:12:11.957 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T20:12:12.415 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:12:12.929 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:12 smithi094 ceph-mon[28050]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:12.950 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:12 smithi094 ceph-mon[28050]: pgmap v446: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:13.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:12 smithi103 ceph-mon[32635]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:13.080 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:12 smithi103 ceph-mon[32635]: pgmap v446: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:13.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:12 smithi204 ceph-mon[32618]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:13.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:12 smithi204 ceph-mon[32618]: pgmap v446: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:14.788 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:14 smithi094 ceph-mon[28050]: pgmap v447: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:14.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.840+0000 7f8009400700 1 -- 172.21.15.94:0/2108404871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8004107260 msgr2=0x7f800410db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:14.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.840+0000 7f8009400700 1 --2- 172.21.15.94:0/2108404871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8004107260 0x7f800410db10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0099c0 tx=0x7f7ffc009cd0 comp rx=0 tx=0).stop 2024-04-26T20:12:14.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.840+0000 7f8009400700 1 -- 172.21.15.94:0/2108404871 shutdown_connections 2024-04-26T20:12:14.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.840+0000 7f8009400700 1 --2- 172.21.15.94:0/2108404871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8004107260 0x7f800410db10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:14.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.840+0000 7f8009400700 1 --2- 172.21.15.94:0/2108404871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80041056c0 0x7f8004105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:14.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.840+0000 7f8009400700 1 --2- 172.21.15.94:0/2108404871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80041068c0 0x7f8004106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:14.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.840+0000 7f8009400700 1 -- 172.21.15.94:0/2108404871 >> 172.21.15.94:0/2108404871 conn(0x7f8004100c80 msgr2=0x7f80041030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:14.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.840+0000 7f8009400700 1 -- 172.21.15.94:0/2108404871 shutdown_connections 2024-04-26T20:12:14.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.840+0000 7f8009400700 1 -- 172.21.15.94:0/2108404871 wait complete. 2024-04-26T20:12:14.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.841+0000 7f8009400700 1 Processor -- start 2024-04-26T20:12:14.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8009400700 1 -- start start 2024-04-26T20:12:14.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8009400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80041056c0 0x7f800419b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:14.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8009400700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80041068c0 0x7f800419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8009400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8004107260 0x7f80041a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8009400700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f80041111b0 con 0x7f80041056c0 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8009400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8004111030 con 0x7f8004107260 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8009400700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8004111330 con 0x7f80041068c0 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8002800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80041056c0 0x7f800419b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8002800700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80041056c0 0x7f800419b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45846/0 (socket says 172.21.15.94:45846) 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8002800700 1 -- 172.21.15.94:0/1294323842 learned_addr learned my addr 172.21.15.94:0/1294323842 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.842+0000 7f8003200700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8004107260 0x7f80041a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.843+0000 7f8001e00700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80041068c0 0x7f800419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.843+0000 7f8002800700 1 -- 172.21.15.94:0/1294323842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80041068c0 msgr2=0x7f800419bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.843+0000 7f8002800700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80041068c0 0x7f800419bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.843+0000 7f8002800700 1 -- 172.21.15.94:0/1294323842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8004107260 msgr2=0x7f80041a2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:14.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.843+0000 7f8002800700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8004107260 0x7f80041a2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:14.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.843+0000 7f8002800700 1 -- 172.21.15.94:0/1294323842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80041a33d0 con 0x7f80041056c0 2024-04-26T20:12:14.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.843+0000 7f8002800700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80041056c0 0x7f800419b770 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009f20 tx=0x7f7ff400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:14.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.843+0000 7f7ffb400700 1 -- 172.21.15.94:0/1294323842 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4003a00 con 0x7f80041056c0 2024-04-26T20:12:14.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.843+0000 7f7ffb400700 1 -- 172.21.15.94:0/1294323842 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ff400ce70 con 0x7f80041056c0 2024-04-26T20:12:14.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.844+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80041a36c0 con 0x7f80041056c0 2024-04-26T20:12:14.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.844+0000 7f7ffb400700 1 -- 172.21.15.94:0/1294323842 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4004710 con 0x7f80041056c0 2024-04-26T20:12:14.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.844+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80041a3c00 con 0x7f80041056c0 2024-04-26T20:12:14.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.845+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f800406a110 con 0x7f80041056c0 2024-04-26T20:12:14.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.848+0000 7f7ffb400700 1 -- 172.21.15.94:0/1294323842 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7ff4012050 con 0x7f80041056c0 2024-04-26T20:12:14.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.850+0000 7f7ffb400700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7fe80a3d50 0x7f7fe80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.850+0000 7f7ffb400700 1 -- 172.21.15.94:0/1294323842 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f7ff40d09f0 con 0x7f80041056c0 2024-04-26T20:12:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.850+0000 7f8001e00700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7fe80a3d50 0x7f7fe80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:14.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.851+0000 7f8001e00700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7fe80a3d50 0x7f7fe80a6210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009c60 tx=0x7f7ff0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:14.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:14.853+0000 7f7ffb400700 1 -- 172.21.15.94:0/1294323842 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7ff4010040 con 0x7f80041056c0 2024-04-26T20:12:15.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:14 smithi103 ceph-mon[32635]: pgmap v447: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:15.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.128+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f8004064030 con 0x7f80041056c0 2024-04-26T20:12:15.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.129+0000 7f7ffb400700 1 -- 172.21.15.94:0/1294323842 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3056 (secure 0 0 0) 0x7f7ff40048b0 con 0x7f80041056c0 2024-04-26T20:12:15.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:14 smithi204 ceph-mon[32618]: pgmap v447: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:e24 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:legacy client fscid: 1 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:Filesystem 'cephfs' (1) 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:fs_name cephfs 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 24 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:created 2024-04-26T20:08:18.399846+0000 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:modified 2024-04-26T20:09:27.103941+0000 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:tableserver 0 2024-04-26T20:12:15.132 INFO:teuthology.orchestra.run.smithi094.stdout:root 0 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:session_timeout 300 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:session_autoclose 300 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:max_file_size 1099511627776 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:max_xattr_size 65536 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:required_client_features {} 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:last_failure 0 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:last_failure_osd_epoch 0 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:max_mds 3 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:in 0,1,2 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:up {0=24575,1=24562,2=24541} 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:failed 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:damaged 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:stopped 2024-04-26T20:12:15.133 INFO:teuthology.orchestra.run.smithi094.stdout:data_pools [3,4] 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:metadata_pool 2 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:inline_data disabled 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:balancer 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:bal_rank_mask -1 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:standby_count_wanted 0 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:qdb_cluster leader: 24575 members: 24575,24562,24541 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.b{0:24575} state up:active seq 6 addr [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.h{0:24530} state up:standby-replay seq 1 addr [v2:172.21.15.103:6836/3697356689,v1:172.21.15.103:6837/3697356689] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.a{1:24562} state up:active seq 22 addr [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.c{1:24520} state up:standby-replay seq 1 addr [v2:172.21.15.204:6836/820565196,v1:172.21.15.204:6837/820565196] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.i{2:24541} state up:active seq 25 addr [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.134 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.f{2:24508} state up:standby-replay seq 1 addr [v2:172.21.15.204:6834/4163398350,v1:172.21.15.204:6835/4163398350] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout:Standby daemons: 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.j{-1:14391} state up:standby seq 1 addr [v2:172.21.15.94:6834/3305420604,v1:172.21.15.94:6835/3305420604] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.d{-1:14409} state up:standby seq 1 addr [v2:172.21.15.94:6836/3734002100,v1:172.21.15.94:6837/3734002100] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.g{-1:14451} state up:standby seq 1 addr [v2:172.21.15.94:6838/2249987305,v1:172.21.15.94:6839/2249987305] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.l{-1:24445} state up:standby seq 1 addr [v2:172.21.15.204:6832/3414557921,v1:172.21.15.204:6833/3414557921] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.k{-1:24452} state up:standby seq 1 addr [v2:172.21.15.103:6832/695386981,v1:172.21.15.103:6833/695386981] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.135 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.e{-1:24500} state up:standby seq 1 addr [v2:172.21.15.103:6834/4015617666,v1:172.21.15.103:6835/4015617666] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:12:15.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.133+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7fe80a3d50 msgr2=0x7f7fe80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:15.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.133+0000 7f8009400700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7fe80a3d50 0x7f7fe80a6210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009c60 tx=0x7f7ff0009260 comp rx=0 tx=0).stop 2024-04-26T20:12:15.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.133+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80041056c0 msgr2=0x7f800419b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:15.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.133+0000 7f8009400700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80041056c0 0x7f800419b770 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009f20 tx=0x7f7ff400c3a0 comp rx=0 tx=0).stop 2024-04-26T20:12:15.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.134+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 shutdown_connections 2024-04-26T20:12:15.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.134+0000 7f8009400700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f7fe80a3d50 0x7f7fe80a6210 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:15.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.134+0000 7f8009400700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8004107260 0x7f80041a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:15.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.134+0000 7f8009400700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80041056c0 0x7f800419b770 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:15.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.134+0000 7f8009400700 1 --2- 172.21.15.94:0/1294323842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80041068c0 0x7f800419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:15.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.134+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 >> 172.21.15.94:0/1294323842 conn(0x7f8004100c80 msgr2=0x7f8004109b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:15.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.134+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 shutdown_connections 2024-04-26T20:12:15.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:15.134+0000 7f8009400700 1 -- 172.21.15.94:0/1294323842 wait complete. 2024-04-26T20:12:15.137 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-26T20:12:15.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:15 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1294323842' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T20:12:16.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:15 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1294323842' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T20:12:16.050 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T20:12:16.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:15 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1294323842' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T20:12:16.481 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:12:16.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:16 smithi094 ceph-mon[28050]: pgmap v448: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:17.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:16 smithi103 ceph-mon[32635]: pgmap v448: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:16 smithi204 ceph-mon[32618]: pgmap v448: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:18.882 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:18 smithi094 ceph-mon[28050]: pgmap v449: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:18.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.923+0000 7f8edfe00700 1 -- 172.21.15.94:0/1719970772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee01045a0 msgr2=0x7f8ee0104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:18.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.923+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1719970772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee01045a0 0x7f8ee0104a00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80099c0 tx=0x7f8ec8009cd0 comp rx=0 tx=0).stop 2024-04-26T20:12:18.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.923+0000 7f8edfe00700 1 -- 172.21.15.94:0/1719970772 shutdown_connections 2024-04-26T20:12:18.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.923+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1719970772 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ee0104f40 0x7f8ee010b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:18.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.923+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1719970772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee01033a0 0x7f8ee01037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:18.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.923+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1719970772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee01045a0 0x7f8ee0104a00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:18.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.923+0000 7f8edfe00700 1 -- 172.21.15.94:0/1719970772 >> 172.21.15.94:0/1719970772 conn(0x7f8ee00fe960 msgr2=0x7f8ee0100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:18.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.924+0000 7f8edfe00700 1 -- 172.21.15.94:0/1719970772 shutdown_connections 2024-04-26T20:12:18.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.924+0000 7f8edfe00700 1 -- 172.21.15.94:0/1719970772 wait complete. 2024-04-26T20:12:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.925+0000 7f8edfe00700 1 Processor -- start 2024-04-26T20:12:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.925+0000 7f8edfe00700 1 -- start start 2024-04-26T20:12:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.925+0000 7f8edfe00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ee01033a0 0x7f8ee00771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.925+0000 7f8edfe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee01045a0 0x7f8ee0075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.925+0000 7f8edfe00700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee0104f40 0x7f8ee0075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.925+0000 7f8edfe00700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010ee30 con 0x7f8ee0104f40 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.925+0000 7f8edfe00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010ecb0 con 0x7f8ee01045a0 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.925+0000 7f8edfe00700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010efb0 con 0x7f8ee01033a0 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.925+0000 7f8edf400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee0104f40 0x7f8ee0075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8ed6000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee01045a0 0x7f8ee0075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8edea00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ee01033a0 0x7f8ee00771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8ed6000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee01045a0 0x7f8ee0075810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:35408/0 (socket says 172.21.15.94:35408) 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8edf400700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee0104f40 0x7f8ee0075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45856/0 (socket says 172.21.15.94:45856) 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8ed6000700 1 -- 172.21.15.94:0/1642020822 learned_addr learned my addr 172.21.15.94:0/1642020822 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8edea00700 1 -- 172.21.15.94:0/1642020822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee01045a0 msgr2=0x7f8ee0075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8edea00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee01045a0 0x7f8ee0075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8edea00700 1 -- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee0104f40 msgr2=0x7f8ee0075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8edea00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee0104f40 0x7f8ee0075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8edea00700 1 -- 172.21.15.94:0/1642020822 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee0076600 con 0x7f8ee01033a0 2024-04-26T20:12:18.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8edf400700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee0104f40 0x7f8ee0075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8ed6000700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee01045a0 0x7f8ee0075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.926+0000 7f8edea00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ee01033a0 0x7f8ee00771c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000e9c0 tx=0x7f8ed000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.927+0000 7f8ed7e00700 1 -- 172.21.15.94:0/1642020822 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed000cb70 con 0x7f8ee01033a0 2024-04-26T20:12:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.927+0000 7f8ed7e00700 1 -- 172.21.15.94:0/1642020822 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ed000ee60 con 0x7f8ee01033a0 2024-04-26T20:12:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.927+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee01a3600 con 0x7f8ee01033a0 2024-04-26T20:12:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.927+0000 7f8ed7e00700 1 -- 172.21.15.94:0/1642020822 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0018680 con 0x7f8ee01033a0 2024-04-26T20:12:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.927+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee01a3b40 con 0x7f8ee01033a0 2024-04-26T20:12:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.928+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee006a110 con 0x7f8ee01033a0 2024-04-26T20:12:18.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.930+0000 7f8ed7e00700 1 -- 172.21.15.94:0/1642020822 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8ed001f020 con 0x7f8ee01033a0 2024-04-26T20:12:18.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.931+0000 7f8ed7e00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8ec40a3d50 0x7f8ec40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:18.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.931+0000 7f8ed7e00700 1 -- 172.21.15.94:0/1642020822 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f8ed00d0720 con 0x7f8ee01033a0 2024-04-26T20:12:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.931+0000 7f8ed6000700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8ec40a3d50 0x7f8ec40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.932+0000 7f8ed6000700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8ec40a3d50 0x7f8ec40a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80036f0 tx=0x7f8ec80037c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:18.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:18.935+0000 7f8ed7e00700 1 -- 172.21.15.94:0/1642020822 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8ed009cb70 con 0x7f8ee01033a0 2024-04-26T20:12:19.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:18 smithi103 ceph-mon[32635]: pgmap v449: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:19.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:18 smithi204 ceph-mon[32618]: pgmap v449: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.162+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f8ee002e490 con 0x7f8ee01033a0 2024-04-26T20:12:19.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.164+0000 7f8ed7e00700 1 -- 172.21.15.94:0/1642020822 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v96) ==== 56+0+5718 (secure 0 0 0) 0x7f8ed009c470 con 0x7f8ee01033a0 2024-04-26T20:12:19.166 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 96 2024-04-26T20:12:19.166 INFO:teuthology.orchestra.run.smithi094.stdout:fsid 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:created 2024-04-26T19:58:57.906790+0000 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:modified 2024-04-26T20:10:13.755106+0000 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:crush_version 27 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:full_ratio 0.95 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:backfillfull_ratio 0.9 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:nearfull_ratio 0.85 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:require_min_compat_client luminous 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:min_compat_client luminous 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:require_osd_release squid 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:stretch_mode_enabled false 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-26T20:12:19.167 INFO:teuthology.orchestra.run.smithi094.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 92 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:max_osd 12 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.0 up in weight 1 up_from 8 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6802/3381923901,v1:172.21.15.94:6803/3381923901] [v2:172.21.15.94:6804/3381923901,v1:172.21.15.94:6805/3381923901] exists,up b6d79ff6-dda1-4462-b2e6-eac0c688fef7 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.1 up in weight 1 up_from 13 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6800/1497163956,v1:172.21.15.103:6801/1497163956] [v2:172.21.15.103:6802/1497163956,v1:172.21.15.103:6803/1497163956] exists,up eef6efa3-ce90-4d2d-8fb8-4a2665258434 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6800/1401040530,v1:172.21.15.204:6801/1401040530] [v2:172.21.15.204:6802/1401040530,v1:172.21.15.204:6803/1401040530] exists,up 939eb12b-359a-4c5e-8e2f-c94e6512050e 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.3 up in weight 1 up_from 25 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6810/2184382995,v1:172.21.15.94:6811/2184382995] [v2:172.21.15.94:6812/2184382995,v1:172.21.15.94:6813/2184382995] exists,up aa4d0799-6fb8-439e-a987-91a977ac9172 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.4 up in weight 1 up_from 33 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6808/3296783557,v1:172.21.15.103:6809/3296783557] [v2:172.21.15.103:6810/3296783557,v1:172.21.15.103:6811/3296783557] exists,up b5d3851f-30c6-4ec7-a8da-0b949c46c6c8 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.5 up in weight 1 up_from 38 up_thru 95 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6808/1603790096,v1:172.21.15.204:6809/1603790096] [v2:172.21.15.204:6810/1603790096,v1:172.21.15.204:6811/1603790096] exists,up c3e5aa68-5a1c-449c-b613-e00d934eb78e 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.6 up in weight 1 up_from 46 up_thru 95 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6818/4225405603,v1:172.21.15.94:6819/4225405603] [v2:172.21.15.94:6820/4225405603,v1:172.21.15.94:6821/4225405603] exists,up 438ff921-8607-404a-bffd-7234940076bd 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.7 up in weight 1 up_from 51 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6816/1379914194,v1:172.21.15.103:6817/1379914194] [v2:172.21.15.103:6818/1379914194,v1:172.21.15.103:6819/1379914194] exists,up e1df6748-fae6-4f37-806d-a6093ec9ca60 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.8 up in weight 1 up_from 59 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6816/1919172767,v1:172.21.15.204:6817/1919172767] [v2:172.21.15.204:6818/1919172767,v1:172.21.15.204:6819/1919172767] exists,up 733fdc3e-cdc8-4f38-afd9-6bd2012145af 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.9 up in weight 1 up_from 67 up_thru 95 down_at 0 last_clean_interval [0,0) [v2:172.21.15.94:6826/4013917902,v1:172.21.15.94:6827/4013917902] [v2:172.21.15.94:6828/4013917902,v1:172.21.15.94:6829/4013917902] exists,up 6bdd2b36-7b98-470b-b0c1-659406d966b7 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.10 up in weight 1 up_from 72 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6824/2183890761,v1:172.21.15.103:6825/2183890761] [v2:172.21.15.103:6826/2183890761,v1:172.21.15.103:6827/2183890761] exists,up 95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:osd.11 up in weight 1 up_from 80 up_thru 95 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6824/1860743299,v1:172.21.15.204:6825/1860743299] [v2:172.21.15.204:6826/1860743299,v1:172.21.15.204:6827/1860743299] exists,up dc362076-e2fb-4994-97c6-08e2cee8e0e0 2024-04-26T20:12:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 2.b [8,11] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 2.e [2,4] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 2.1b [2,1] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 2.1f [7,11] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.6 [3,7] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.11 [5,9] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.1b [5,9] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.1d [4,6] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.24 [4,9] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 3.3a [4,8] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 4.6 [5,8] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 4.c [9,10] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 4.13 [9,8] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:pg_upmap_items 4.18 [9,8] 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/3877259400 expires 2024-04-27T20:00:11.413271+0000 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/3361956945 expires 2024-04-27T20:00:11.413271+0000 2024-04-26T20:12:19.169 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:6801/3651438916 expires 2024-04-27T20:00:11.413271+0000 2024-04-26T20:12:19.170 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:6801/3005317891 expires 2024-04-27T19:59:38.717857+0000 2024-04-26T20:12:19.170 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/1688404742 expires 2024-04-27T19:59:38.717857+0000 2024-04-26T20:12:19.170 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/2041870658 expires 2024-04-27T19:59:38.717857+0000 2024-04-26T20:12:19.170 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/442392134 expires 2024-04-27T19:59:38.717857+0000 2024-04-26T20:12:19.170 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:0/1853030572 expires 2024-04-27T20:00:11.413271+0000 2024-04-26T20:12:19.170 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:6800/3005317891 expires 2024-04-27T19:59:38.717857+0000 2024-04-26T20:12:19.170 INFO:teuthology.orchestra.run.smithi094.stdout:blocklist 172.21.15.94:6800/3651438916 expires 2024-04-27T20:00:11.413271+0000 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.167+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8ec40a3d50 msgr2=0x7f8ec40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.167+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8ec40a3d50 0x7f8ec40a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80036f0 tx=0x7f8ec80037c0 comp rx=0 tx=0).stop 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.167+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ee01033a0 msgr2=0x7f8ee00771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.167+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ee01033a0 0x7f8ee00771c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000e9c0 tx=0x7f8ed000ecd0 comp rx=0 tx=0).stop 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.167+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 shutdown_connections 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.167+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ee0104f40 0x7f8ee0075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.167+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ee01033a0 0x7f8ee00771c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.167+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8ec40a3d50 0x7f8ec40a6210 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.167+0000 7f8edfe00700 1 --2- 172.21.15.94:0/1642020822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee01045a0 0x7f8ee0075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.168+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 >> 172.21.15.94:0/1642020822 conn(0x7f8ee00fe960 msgr2=0x7f8ee01078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.168+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 shutdown_connections 2024-04-26T20:12:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:19.168+0000 7f8edfe00700 1 -- 172.21.15.94:0/1642020822 wait complete. 2024-04-26T20:12:19.912 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:19 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1642020822' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T20:12:19.976 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T20:12:19.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:12:20.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:19 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1642020822' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T20:12:20.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:19 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1642020822' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T20:12:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.200+0000 7f477f200640 1 -- 172.21.15.94:0/433132830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780102750 msgr2=0x7f4780102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.200+0000 7f477f200640 1 --2- 172.21.15.94:0/433132830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780102750 0x7f4780102b70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4770009900 tx=0x7f477002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.200+0000 7f477f200640 1 -- 172.21.15.94:0/433132830 shutdown_connections 2024-04-26T20:12:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.200+0000 7f477f200640 1 --2- 172.21.15.94:0/433132830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4780104300 0x7f478010ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.200+0000 7f477f200640 1 --2- 172.21.15.94:0/433132830 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4780103940 0x7f4780103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.200+0000 7f477f200640 1 --2- 172.21.15.94:0/433132830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780102750 0x7f4780102b70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.201+0000 7f477f200640 1 -- 172.21.15.94:0/433132830 >> 172.21.15.94:0/433132830 conn(0x7f47800fdcc0 msgr2=0x7f4780100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:20.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.201+0000 7f477f200640 1 -- 172.21.15.94:0/433132830 shutdown_connections 2024-04-26T20:12:20.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.201+0000 7f477f200640 1 -- 172.21.15.94:0/433132830 wait complete. 2024-04-26T20:12:20.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f477f200640 1 Processor -- start 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f477f200640 1 -- start start 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f477f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4780102750 0x7f478019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f477f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4780103940 0x7f478019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f477f200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780104300 0x7f47801a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f477f200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f478010e250 con 0x7f4780104300 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f477f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f478010e0d0 con 0x7f4780102750 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f477f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f478010e3d0 con 0x7f4780103940 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f477e800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780104300 0x7f47801a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.202+0000 7f4777e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4780103940 0x7f478019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f477e800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780104300 0x7f47801a2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45880/0 (socket says 172.21.15.94:45880) 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f477e800640 1 -- 172.21.15.94:0/4266882599 learned_addr learned my addr 172.21.15.94:0/4266882599 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f477de00640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4780102750 0x7f478019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f477e800640 1 -- 172.21.15.94:0/4266882599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4780103940 msgr2=0x7f478019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f477e800640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4780103940 0x7f478019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f477e800640 1 -- 172.21.15.94:0/4266882599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4780102750 msgr2=0x7f478019abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f477e800640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4780102750 0x7f478019abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f477e800640 1 -- 172.21.15.94:0/4266882599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47801a27a0 con 0x7f4780104300 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f477e800640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780104300 0x7f47801a2130 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f477800eb10 tx=0x7f477800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.203+0000 7f4776a00640 1 -- 172.21.15.94:0/4266882599 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778004b20 con 0x7f4780104300 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.204+0000 7f4776a00640 1 -- 172.21.15.94:0/4266882599 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4778004cc0 con 0x7f4780104300 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.204+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47801a2a90 con 0x7f4780104300 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.204+0000 7f4776a00640 1 -- 172.21.15.94:0/4266882599 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778010670 con 0x7f4780104300 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.204+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47801a2fe0 con 0x7f4780104300 2024-04-26T20:12:20.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.205+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47801a3270 con 0x7f4780104300 2024-04-26T20:12:20.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.207+0000 7f4776a00640 1 -- 172.21.15.94:0/4266882599 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4778005020 con 0x7f4780104300 2024-04-26T20:12:20.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.208+0000 7f4776a00640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f47640a3cc0 0x7f47640a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:20.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.208+0000 7f4776a00640 1 -- 172.21.15.94:0/4266882599 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f47780d0e50 con 0x7f4780104300 2024-04-26T20:12:20.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.208+0000 7f477de00640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f47640a3cc0 0x7f47640a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:20.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.209+0000 7f477de00640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f47640a3cc0 0x7f47640a6180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4770009230 tx=0x7f47700314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:20.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.213+0000 7f4776a00640 1 -- 172.21.15.94:0/4266882599 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f477809d2a0 con 0x7f4780104300 2024-04-26T20:12:20.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.418+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47801a34e0 con 0x7f4780104300 2024-04-26T20:12:20.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.421+0000 7f4776a00640 1 -- 172.21.15.94:0/4266882599 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24879 (secure 0 0 0) 0x7f477809cba0 con 0x7f4780104300 2024-04-26T20:12:20.423 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:12:20.424 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:10:13.755106+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:12:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f47640a3cc0 msgr2=0x7f47640a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f47640a3cc0 0x7f47640a6180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4770009230 tx=0x7f47700314a0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780104300 msgr2=0x7f47801a2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780104300 0x7f47801a2130 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f477800eb10 tx=0x7f477800c2d0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 shutdown_connections 2024-04-26T20:12:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f47640a3cc0 0x7f47640a6180 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4780104300 0x7f47801a2130 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4780103940 0x7f478019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 --2- 172.21.15.94:0/4266882599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4780102750 0x7f478019abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.424+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 >> 172.21.15.94:0/4266882599 conn(0x7f47800fdcc0 msgr2=0x7f4780074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.425+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 shutdown_connections 2024-04-26T20:12:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.425+0000 7f477f200640 1 -- 172.21.15.94:0/4266882599 wait complete. 2024-04-26T20:12:20.445 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:08:13.904332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:08:15.670247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:08:23.682553+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:12:20.445 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:12:20.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.651+0000 7f020b200640 1 -- 172.21.15.94:0/1556032469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f020c106620 msgr2=0x7f020c10ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.651+0000 7f020b200640 1 --2- 172.21.15.94:0/1556032469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f020c106620 0x7f020c10ceb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0204009ca0 tx=0x7f020402a8b0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.651+0000 7f020b200640 1 -- 172.21.15.94:0/1556032469 shutdown_connections 2024-04-26T20:12:20.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.651+0000 7f020b200640 1 --2- 172.21.15.94:0/1556032469 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f020c106620 0x7f020c10ceb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.651+0000 7f020b200640 1 --2- 172.21.15.94:0/1556032469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f020c105c60 0x7f020c1060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.651+0000 7f020b200640 1 --2- 172.21.15.94:0/1556032469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f020c104a70 0x7f020c104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.651+0000 7f020b200640 1 -- 172.21.15.94:0/1556032469 >> 172.21.15.94:0/1556032469 conn(0x7f020c0fffe0 msgr2=0x7f020c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:20.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.651+0000 7f020b200640 1 -- 172.21.15.94:0/1556032469 shutdown_connections 2024-04-26T20:12:20.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.651+0000 7f020b200640 1 -- 172.21.15.94:0/1556032469 wait complete. 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.652+0000 7f020b200640 1 Processor -- start 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.652+0000 7f020b200640 1 -- start start 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f020b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f020c104a70 0x7f020c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f020b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f020c105c60 0x7f020c19b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f020b200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f020c106620 0x7f020c1a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f020b200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f020c1105f0 con 0x7f020c106620 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f020b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f020c110470 con 0x7f020c105c60 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f020b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f020c110770 con 0x7f020c104a70 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f020a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f020c106620 0x7f020c1a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f020a800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f020c106620 0x7f020c1a2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45900/0 (socket says 172.21.15.94:45900) 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f0209400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f020c105c60 0x7f020c19b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f020a800640 1 -- 172.21.15.94:0/2757863266 learned_addr learned my addr 172.21.15.94:0/2757863266 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:20.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f0209e00640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f020c104a70 0x7f020c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:20.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f0209400640 1 -- 172.21.15.94:0/2757863266 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f020c104a70 msgr2=0x7f020c19ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f0209400640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f020c104a70 0x7f020c19ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f0209400640 1 -- 172.21.15.94:0/2757863266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f020c106620 msgr2=0x7f020c1a2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f0209400640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f020c106620 0x7f020c1a2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.653+0000 7f0209400640 1 -- 172.21.15.94:0/2757863266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020c1a28b0 con 0x7f020c105c60 2024-04-26T20:12:20.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.654+0000 7f0209400640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f020c105c60 0x7f020c19b1b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f01f800e9f0 tx=0x7f01f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:20.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.654+0000 7f0202a00640 1 -- 172.21.15.94:0/2757863266 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f800cd80 con 0x7f020c105c60 2024-04-26T20:12:20.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.654+0000 7f0202a00640 1 -- 172.21.15.94:0/2757863266 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01f80035f0 con 0x7f020c105c60 2024-04-26T20:12:20.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.654+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020c1a2ba0 con 0x7f020c105c60 2024-04-26T20:12:20.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.654+0000 7f0202a00640 1 -- 172.21.15.94:0/2757863266 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8010690 con 0x7f020c105c60 2024-04-26T20:12:20.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.654+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f020c1a3140 con 0x7f020c105c60 2024-04-26T20:12:20.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.655+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f020c10bd90 con 0x7f020c105c60 2024-04-26T20:12:20.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.658+0000 7f0202a00640 1 -- 172.21.15.94:0/2757863266 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f01f80037a0 con 0x7f020c105c60 2024-04-26T20:12:20.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.660+0000 7f0202a00640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f01f00a3cc0 0x7f01f00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:20.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.660+0000 7f0202a00640 1 -- 172.21.15.94:0/2757863266 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f01f80d0db0 con 0x7f020c105c60 2024-04-26T20:12:20.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.660+0000 7f0209e00640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f01f00a3cc0 0x7f01f00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:20.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.660+0000 7f0209e00640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f01f00a3cc0 0x7f01f00a6180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f020c19bb40 tx=0x7f01fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:20.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.662+0000 7f0202a00640 1 -- 172.21.15.94:0/2757863266 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f01f809d200 con 0x7f020c105c60 2024-04-26T20:12:20.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.863+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f020c02dad0 con 0x7f020c105c60 2024-04-26T20:12:20.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.864+0000 7f0202a00640 1 -- 172.21.15.94:0/2757863266 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f01f809cb00 con 0x7f020c105c60 2024-04-26T20:12:20.865 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:12:20.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.866+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f01f00a3cc0 msgr2=0x7f01f00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.866+0000 7f020b200640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f01f00a3cc0 0x7f01f00a6180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f020c19bb40 tx=0x7f01fc005f90 comp rx=0 tx=0).stop 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.866+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f020c105c60 msgr2=0x7f020c19b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.866+0000 7f020b200640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f020c105c60 0x7f020c19b1b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f01f800e9f0 tx=0x7f01f800eeb0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.867+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 shutdown_connections 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.867+0000 7f020b200640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f01f00a3cc0 0x7f01f00a6180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.867+0000 7f020b200640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f020c106620 0x7f020c1a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.867+0000 7f020b200640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f020c105c60 0x7f020c19b1b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.867+0000 7f020b200640 1 --2- 172.21.15.94:0/2757863266 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f020c104a70 0x7f020c19ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.867+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 >> 172.21.15.94:0/2757863266 conn(0x7f020c0fffe0 msgr2=0x7f020c108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.867+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 shutdown_connections 2024-04-26T20:12:20.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:20.867+0000 7f020b200640 1 -- 172.21.15.94:0/2757863266 wait complete. 2024-04-26T20:12:20.886 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:12:21.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:20 smithi103 ceph-mon[32635]: pgmap v450: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:21.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:20 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4266882599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:12:21.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.087+0000 7f52d5600640 1 -- 172.21.15.94:0/2601406223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0076970 msgr2=0x7f52d0074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.087+0000 7f52d5600640 1 --2- 172.21.15.94:0/2601406223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0076970 0x7f52d0074dc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009930 tx=0x7f52c002a420 comp rx=0 tx=0).stop 2024-04-26T20:12:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.087+0000 7f52d5600640 1 -- 172.21.15.94:0/2601406223 shutdown_connections 2024-04-26T20:12:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.087+0000 7f52d5600640 1 --2- 172.21.15.94:0/2601406223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52d010ee30 0x7f52d0111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.087+0000 7f52d5600640 1 --2- 172.21.15.94:0/2601406223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52d0075390 0x7f52d0075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.087+0000 7f52d5600640 1 --2- 172.21.15.94:0/2601406223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0076970 0x7f52d0074dc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.087+0000 7f52d5600640 1 -- 172.21.15.94:0/2601406223 >> 172.21.15.94:0/2601406223 conn(0x7f52d00fddd0 msgr2=0x7f52d0100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.087+0000 7f52d5600640 1 -- 172.21.15.94:0/2601406223 shutdown_connections 2024-04-26T20:12:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.087+0000 7f52d5600640 1 -- 172.21.15.94:0/2601406223 wait complete. 2024-04-26T20:12:21.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.088+0000 7f52d5600640 1 Processor -- start 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.088+0000 7f52d5600640 1 -- start start 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52d5600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0075390 0x7f52d019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52d5600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52d0076970 0x7f52d019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52d5600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52d010ee30 0x7f52d01a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52d5600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f52d01149d0 con 0x7f52d0075390 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52d5600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f52d0114850 con 0x7f52d010ee30 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52d5600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f52d0114b50 con 0x7f52d0076970 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52cfe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0075390 0x7f52d019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52cfe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0075390 0x7f52d019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46258/0 (socket says 172.21.15.94:46258) 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52cfe00640 1 -- 172.21.15.94:0/3771942338 learned_addr learned my addr 172.21.15.94:0/3771942338 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52c7400640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52d0076970 0x7f52d019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52d4c00640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52d010ee30 0x7f52d01a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52cfe00640 1 -- 172.21.15.94:0/3771942338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52d0076970 msgr2=0x7f52d019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52cfe00640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52d0076970 0x7f52d019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52cfe00640 1 -- 172.21.15.94:0/3771942338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52d010ee30 msgr2=0x7f52d01a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52cfe00640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52d010ee30 0x7f52d01a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52cfe00640 1 -- 172.21.15.94:0/3771942338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d01a6cf0 con 0x7f52d0075390 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.089+0000 7f52d4c00640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52d010ee30 0x7f52d01a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.090+0000 7f52cfe00640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0075390 0x7f52d019f0d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f52c0031500 tx=0x7f52c0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.090+0000 7f52cd600640 1 -- 172.21.15.94:0/3771942338 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c003f070 con 0x7f52d0075390 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.090+0000 7f52cd600640 1 -- 172.21.15.94:0/3771942338 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52c0031c50 con 0x7f52d0075390 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.090+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d01a6f80 con 0x7f52d0075390 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.090+0000 7f52cd600640 1 -- 172.21.15.94:0/3771942338 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0035030 con 0x7f52d0075390 2024-04-26T20:12:21.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.090+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d01a73e0 con 0x7f52d0075390 2024-04-26T20:12:21.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.091+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d0105000 con 0x7f52d0075390 2024-04-26T20:12:21.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.093+0000 7f52cd600640 1 -- 172.21.15.94:0/3771942338 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f52c0042070 con 0x7f52d0075390 2024-04-26T20:12:21.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.094+0000 7f52cd600640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52b40a3cc0 0x7f52b40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.095+0000 7f52cd600640 1 -- 172.21.15.94:0/3771942338 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f52c00f10a0 con 0x7f52d0075390 2024-04-26T20:12:21.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.095+0000 7f52c7400640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52b40a3cc0 0x7f52b40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.095+0000 7f52c7400640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52b40a3cc0 0x7f52b40a6180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f52d01a05e0 tx=0x7f52b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:21.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.098+0000 7f52cd600640 1 -- 172.21.15.94:0/3771942338 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f52c00bd4f0 con 0x7f52d0075390 2024-04-26T20:12:21.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:20 smithi204 ceph-mon[32618]: pgmap v450: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:21.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:20 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4266882599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:12:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:20 smithi094 ceph-mon[28050]: pgmap v450: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:20 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4266882599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:12:21.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.301+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f52d002dd20 con 0x7f52d0075390 2024-04-26T20:12:21.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.302+0000 7f52cd600640 1 -- 172.21.15.94:0/3771942338 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f52c00bcdf0 con 0x7f52d0075390 2024-04-26T20:12:21.303 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:12:21.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.304+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52b40a3cc0 msgr2=0x7f52b40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.304+0000 7f52d5600640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52b40a3cc0 0x7f52b40a6180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f52d01a05e0 tx=0x7f52b8005f90 comp rx=0 tx=0).stop 2024-04-26T20:12:21.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.304+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0075390 msgr2=0x7f52d019f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.304+0000 7f52d5600640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0075390 0x7f52d019f0d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f52c0031500 tx=0x7f52c0003c10 comp rx=0 tx=0).stop 2024-04-26T20:12:21.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.305+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 shutdown_connections 2024-04-26T20:12:21.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.305+0000 7f52d5600640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52b40a3cc0 0x7f52b40a6180 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.305+0000 7f52d5600640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52d010ee30 0x7f52d01a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.305+0000 7f52d5600640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52d0076970 0x7f52d019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.305+0000 7f52d5600640 1 --2- 172.21.15.94:0/3771942338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52d0075390 0x7f52d019f0d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.305+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 >> 172.21.15.94:0/3771942338 conn(0x7f52d00fddd0 msgr2=0x7f52d01021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:21.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.305+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 shutdown_connections 2024-04-26T20:12:21.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.305+0000 7f52d5600640 1 -- 172.21.15.94:0/3771942338 wait complete. 2024-04-26T20:12:21.324 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:12:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.522+0000 7fd105000640 1 -- 172.21.15.94:0/896407669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd10010a180 msgr2=0x7fd10010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.522+0000 7fd105000640 1 --2- 172.21.15.94:0/896407669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd10010a180 0x7fd10010a600 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8009900 tx=0x7fd0e802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.522+0000 7fd105000640 1 -- 172.21.15.94:0/896407669 shutdown_connections 2024-04-26T20:12:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.522+0000 7fd105000640 1 --2- 172.21.15.94:0/896407669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd10010ab40 0x7fd1001113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.522+0000 7fd105000640 1 --2- 172.21.15.94:0/896407669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd10010a180 0x7fd10010a600 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.522+0000 7fd105000640 1 --2- 172.21.15.94:0/896407669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd100108f90 0x7fd1001093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.522+0000 7fd105000640 1 -- 172.21.15.94:0/896407669 >> 172.21.15.94:0/896407669 conn(0x7fd100077f30 msgr2=0x7fd10007a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.522+0000 7fd105000640 1 -- 172.21.15.94:0/896407669 shutdown_connections 2024-04-26T20:12:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.522+0000 7fd105000640 1 -- 172.21.15.94:0/896407669 wait complete. 2024-04-26T20:12:21.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.523+0000 7fd105000640 1 Processor -- start 2024-04-26T20:12:21.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.523+0000 7fd105000640 1 -- start start 2024-04-26T20:12:21.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.523+0000 7fd105000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd100108f90 0x7fd10019f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.523+0000 7fd105000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd10010a180 0x7fd10019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.523+0000 7fd105000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd10010ab40 0x7fd1001a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.523+0000 7fd105000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fd100114b20 con 0x7fd10010ab40 2024-04-26T20:12:21.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.523+0000 7fd105000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001149a0 con 0x7fd10010a180 2024-04-26T20:12:21.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.523+0000 7fd105000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd100114ca0 con 0x7fd100108f90 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fe800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd10010ab40 0x7fd1001a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd100108f90 0x7fd10019f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd100108f90 0x7fd10019f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:50428/0 (socket says 172.21.15.94:50428) 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fe800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd10010ab40 0x7fd1001a6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46280/0 (socket says 172.21.15.94:46280) 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fde00640 1 -- 172.21.15.94:0/3715091981 learned_addr learned my addr 172.21.15.94:0/3715091981 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0f7e00640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd10010a180 0x7fd10019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fde00640 1 -- 172.21.15.94:0/3715091981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd10010a180 msgr2=0x7fd10019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fde00640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd10010a180 0x7fd10019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fde00640 1 -- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd10010ab40 msgr2=0x7fd1001a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fde00640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd10010ab40 0x7fd1001a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fde00640 1 -- 172.21.15.94:0/3715091981 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1001a6dd0 con 0x7fd100108f90 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fe800640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd10010ab40 0x7fd1001a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:21.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.524+0000 7fd0fde00640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd100108f90 0x7fd10019f1b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000e9c0 tx=0x7fd0f000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.525+0000 7fd0f6a00640 1 -- 172.21.15.94:0/3715091981 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f000cde0 con 0x7fd100108f90 2024-04-26T20:12:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.525+0000 7fd0f6a00640 1 -- 172.21.15.94:0/3715091981 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0f00108e0 con 0x7fd100108f90 2024-04-26T20:12:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.525+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1001a70c0 con 0x7fd100108f90 2024-04-26T20:12:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.525+0000 7fd0f6a00640 1 -- 172.21.15.94:0/3715091981 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f00188d0 con 0x7fd100108f90 2024-04-26T20:12:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.525+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1001a75e0 con 0x7fd100108f90 2024-04-26T20:12:21.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.526+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd100110300 con 0x7fd100108f90 2024-04-26T20:12:21.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.528+0000 7fd0f6a00640 1 -- 172.21.15.94:0/3715091981 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd0f001f020 con 0x7fd100108f90 2024-04-26T20:12:21.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.530+0000 7fd0f6a00640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd0e40a3c40 0x7fd0e40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.530+0000 7fd0f6a00640 1 -- 172.21.15.94:0/3715091981 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fd0f00d0d60 con 0x7fd100108f90 2024-04-26T20:12:21.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.530+0000 7fd0f7e00640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd0e40a3c40 0x7fd0e40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.531+0000 7fd0f7e00640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd0e40a3c40 0x7fd0e40a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8031500 tx=0x7fd0e8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:21.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.532+0000 7fd0f6a00640 1 -- 172.21.15.94:0/3715091981 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd0f009d1b0 con 0x7fd100108f90 2024-04-26T20:12:21.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.730+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd100063530 con 0x7fd100108f90 2024-04-26T20:12:21.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.731+0000 7fd0f6a00640 1 -- 172.21.15.94:0/3715091981 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7fd0f00bd0f0 con 0x7fd100108f90 2024-04-26T20:12:21.732 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.733+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd0e40a3c40 msgr2=0x7fd0e40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.733+0000 7fd105000640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd0e40a3c40 0x7fd0e40a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8031500 tx=0x7fd0e8005a20 comp rx=0 tx=0).stop 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.733+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd100108f90 msgr2=0x7fd10019f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.733+0000 7fd105000640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd100108f90 0x7fd10019f1b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000e9c0 tx=0x7fd0f000ee80 comp rx=0 tx=0).stop 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.733+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 shutdown_connections 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.733+0000 7fd105000640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fd0e40a3c40 0x7fd0e40a6100 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.733+0000 7fd105000640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd10010ab40 0x7fd1001a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.733+0000 7fd105000640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd10010a180 0x7fd10019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.733+0000 7fd105000640 1 --2- 172.21.15.94:0/3715091981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd100108f90 0x7fd10019f1b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.734+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 >> 172.21.15.94:0/3715091981 conn(0x7fd100077f30 msgr2=0x7fd10010d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:21.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.734+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 shutdown_connections 2024-04-26T20:12:21.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.734+0000 7fd105000640 1 -- 172.21.15.94:0/3715091981 wait complete. 2024-04-26T20:12:21.753 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:12:21.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.951+0000 7fa4c4c00640 1 -- 172.21.15.94:0/2893374322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0102860 msgr2=0x7fa4c0102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.951+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/2893374322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0102860 0x7fa4c0102c80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009ca0 tx=0x7fa4b802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.951+0000 7fa4c4c00640 1 -- 172.21.15.94:0/2893374322 shutdown_connections 2024-04-26T20:12:21.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.951+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/2893374322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4c0104410 0x7fa4c010aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.951+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/2893374322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c0103a50 0x7fa4c0103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.951+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/2893374322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0102860 0x7fa4c0102c80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.951+0000 7fa4c4c00640 1 -- 172.21.15.94:0/2893374322 >> 172.21.15.94:0/2893374322 conn(0x7fa4c00fddd0 msgr2=0x7fa4c0100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:21.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.951+0000 7fa4c4c00640 1 -- 172.21.15.94:0/2893374322 shutdown_connections 2024-04-26T20:12:21.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.951+0000 7fa4c4c00640 1 -- 172.21.15.94:0/2893374322 wait complete. 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.953+0000 7fa4c4c00640 1 Processor -- start 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.953+0000 7fa4c4c00640 1 -- start start 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.953+0000 7fa4c4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4c0102860 0x7fa4c019acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.953+0000 7fa4c4c00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0103a50 0x7fa4c019b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.953+0000 7fa4c4c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c0104410 0x7fa4c01a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.953+0000 7fa4c4c00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c010e390 con 0x7fa4c0103a50 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.953+0000 7fa4c4c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c010e210 con 0x7fa4c0104410 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.953+0000 7fa4c4c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c010e510 con 0x7fa4c0102860 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.953+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0103a50 0x7fa4c019b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0103a50 0x7fa4c019b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46288/0 (socket says 172.21.15.94:46288) 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bea00640 1 -- 172.21.15.94:0/535996027 learned_addr learned my addr 172.21.15.94:0/535996027 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bfe00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c0104410 0x7fa4c01a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bf400640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4c0102860 0x7fa4c019acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bea00640 1 -- 172.21.15.94:0/535996027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4c0102860 msgr2=0x7fa4c019acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bea00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4c0102860 0x7fa4c019acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bea00640 1 -- 172.21.15.94:0/535996027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c0104410 msgr2=0x7fa4c01a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bea00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c0104410 0x7fa4c01a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bea00640 1 -- 172.21.15.94:0/535996027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c01a2900 con 0x7fa4c0103a50 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bf400640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4c0102860 0x7fa4c019acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4bea00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0103a50 0x7fa4c019b230 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800ea60 tx=0x7fa4a800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4b7e00640 1 -- 172.21.15.94:0/535996027 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a800cde0 con 0x7fa4c0103a50 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.954+0000 7fa4b7e00640 1 -- 172.21.15.94:0/535996027 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4a80035f0 con 0x7fa4c0103a50 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.955+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c01a2bf0 con 0x7fa4c0103a50 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.955+0000 7fa4b7e00640 1 -- 172.21.15.94:0/535996027 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8010690 con 0x7fa4c0103a50 2024-04-26T20:12:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.955+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c01a3140 con 0x7fa4c0103a50 2024-04-26T20:12:21.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.956+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c01a3370 con 0x7fa4c0103a50 2024-04-26T20:12:21.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.958+0000 7fa4b7e00640 1 -- 172.21.15.94:0/535996027 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa4a80037a0 con 0x7fa4c0103a50 2024-04-26T20:12:21.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.959+0000 7fa4b7e00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa4a40a3c40 0x7fa4a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:21.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.959+0000 7fa4b7e00640 1 -- 172.21.15.94:0/535996027 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fa4a80d0d70 con 0x7fa4c0103a50 2024-04-26T20:12:21.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.959+0000 7fa4bf400640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa4a40a3c40 0x7fa4a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:21.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.960+0000 7fa4bf400640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa4a40a3c40 0x7fa4a40a6100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006010 tx=0x7fa4b8005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:21.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:21.964+0000 7fa4b7e00640 1 -- 172.21.15.94:0/535996027 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa4a809d1c0 con 0x7fa4c0103a50 2024-04-26T20:12:22.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:21 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2757863266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:21 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3771942338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:21 smithi103 ceph-mon[32635]: pgmap v451: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:22.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:21 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3715091981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:21 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2757863266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:21 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3771942338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:21 smithi204 ceph-mon[32618]: pgmap v451: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:22.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:21 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3715091981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.165+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa4c01a3540 con 0x7fa4c0103a50 2024-04-26T20:12:22.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.166+0000 7fa4b7e00640 1 -- 172.21.15.94:0/535996027 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7fa4a809cac0 con 0x7fa4c0103a50 2024-04-26T20:12:22.168 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:12:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa4a40a3c40 msgr2=0x7fa4a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa4a40a3c40 0x7fa4a40a6100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006010 tx=0x7fa4b8005ef0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0103a50 msgr2=0x7fa4c019b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0103a50 0x7fa4c019b230 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800ea60 tx=0x7fa4a800ef20 comp rx=0 tx=0).stop 2024-04-26T20:12:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 shutdown_connections 2024-04-26T20:12:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fa4a40a3c40 0x7fa4a40a6100 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c0104410 0x7fa4c01a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4c0103a50 0x7fa4c019b230 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 --2- 172.21.15.94:0/535996027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4c0102860 0x7fa4c019acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 >> 172.21.15.94:0/535996027 conn(0x7fa4c00fddd0 msgr2=0x7fa4c0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 shutdown_connections 2024-04-26T20:12:22.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.169+0000 7fa4c4c00640 1 -- 172.21.15.94:0/535996027 wait complete. 2024-04-26T20:12:22.187 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:21 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2757863266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.187 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:21 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3771942338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.188 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:21 smithi094 ceph-mon[28050]: pgmap v451: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:22.188 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:21 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3715091981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:12:22.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.393+0000 7f32f2400640 1 -- 172.21.15.94:0/3293826526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32ec105b50 msgr2=0x7f32ec105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.393+0000 7f32f2400640 1 --2- 172.21.15.94:0/3293826526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32ec105b50 0x7f32ec105fd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009860 tx=0x7f32d802a340 comp rx=0 tx=0).stop 2024-04-26T20:12:22.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.394+0000 7f32f2400640 1 -- 172.21.15.94:0/3293826526 shutdown_connections 2024-04-26T20:12:22.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.394+0000 7f32f2400640 1 --2- 172.21.15.94:0/3293826526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32ec106510 0x7f32ec10cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.394+0000 7f32f2400640 1 --2- 172.21.15.94:0/3293826526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32ec105b50 0x7f32ec105fd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.394+0000 7f32f2400640 1 --2- 172.21.15.94:0/3293826526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec104960 0x7f32ec104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.394+0000 7f32f2400640 1 -- 172.21.15.94:0/3293826526 >> 172.21.15.94:0/3293826526 conn(0x7f32ec0ffeb0 msgr2=0x7f32ec102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:22.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.394+0000 7f32f2400640 1 -- 172.21.15.94:0/3293826526 shutdown_connections 2024-04-26T20:12:22.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.394+0000 7f32f2400640 1 -- 172.21.15.94:0/3293826526 wait complete. 2024-04-26T20:12:22.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.395+0000 7f32f2400640 1 Processor -- start 2024-04-26T20:12:22.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.395+0000 7f32f2400640 1 -- start start 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32f2400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32ec104960 0x7f32ec19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32f2400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec105b50 0x7f32ec19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32f2400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32ec106510 0x7f32ec1a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32f2400640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec110490 con 0x7f32ec105b50 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32f2400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec110310 con 0x7f32ec106510 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32f2400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec110610 con 0x7f32ec104960 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32ebe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec105b50 0x7f32ec19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32f1000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32ec104960 0x7f32ec19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32ebe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec105b50 0x7f32ec19b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46306/0 (socket says 172.21.15.94:46306) 2024-04-26T20:12:22.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32ebe00640 1 -- 172.21.15.94:0/260099894 learned_addr learned my addr 172.21.15.94:0/260099894 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32f1a00640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32ec106510 0x7f32ec1a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32ebe00640 1 -- 172.21.15.94:0/260099894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32ec104960 msgr2=0x7f32ec19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32ebe00640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32ec104960 0x7f32ec19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32ebe00640 1 -- 172.21.15.94:0/260099894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32ec106510 msgr2=0x7f32ec1a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32ebe00640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32ec106510 0x7f32ec1a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32ebe00640 1 -- 172.21.15.94:0/260099894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ec1a2780 con 0x7f32ec105b50 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.396+0000 7f32f1a00640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32ec106510 0x7f32ec1a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.397+0000 7f32ebe00640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec105b50 0x7f32ec19b0a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f32d80057c0 tx=0x7f32d8032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.397+0000 7f32e9600640 1 -- 172.21.15.94:0/260099894 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8037040 con 0x7f32ec105b50 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.397+0000 7f32e9600640 1 -- 172.21.15.94:0/260099894 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32d8030bd0 con 0x7f32ec105b50 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.397+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ec1a2a10 con 0x7f32ec105b50 2024-04-26T20:12:22.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.397+0000 7f32e9600640 1 -- 172.21.15.94:0/260099894 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8005280 con 0x7f32ec105b50 2024-04-26T20:12:22.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.397+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ec1a2ef0 con 0x7f32ec105b50 2024-04-26T20:12:22.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.398+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32ec10bd50 con 0x7f32ec105b50 2024-04-26T20:12:22.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.400+0000 7f32e9600640 1 -- 172.21.15.94:0/260099894 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f32d8043020 con 0x7f32ec105b50 2024-04-26T20:12:22.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.402+0000 7f32e9600640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32d40a3cc0 0x7f32d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:22.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.402+0000 7f32e9600640 1 -- 172.21.15.94:0/260099894 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f32d80efed0 con 0x7f32ec105b50 2024-04-26T20:12:22.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.403+0000 7f32f1000640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32d40a3cc0 0x7f32d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:22.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.403+0000 7f32f1000640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32d40a3cc0 0x7f32d40a6180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f32ec1059b0 tx=0x7f32e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:22.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.406+0000 7f32e9600640 1 -- 172.21.15.94:0/260099894 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f32d80bc430 con 0x7f32ec105b50 2024-04-26T20:12:22.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.649+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32ec02dce0 con 0x7f32ec105b50 2024-04-26T20:12:22.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.650+0000 7f32e9600640 1 -- 172.21.15.94:0/260099894 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11878 (secure 0 0 0) 0x7f32d80bbd30 con 0x7f32ec105b50 2024-04-26T20:12:22.652 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:12:22.652 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:09:27.103941+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24562,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24562,24541]},"id":1}]} 2024-04-26T20:12:22.652 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.653+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32d40a3cc0 msgr2=0x7f32d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.653+0000 7f32f2400640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32d40a3cc0 0x7f32d40a6180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f32ec1059b0 tx=0x7f32e0005f90 comp rx=0 tx=0).stop 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.653+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec105b50 msgr2=0x7f32ec19b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.653+0000 7f32f2400640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec105b50 0x7f32ec19b0a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f32d80057c0 tx=0x7f32d8032da0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.654+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 shutdown_connections 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.654+0000 7f32f2400640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32d40a3cc0 0x7f32d40a6180 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.654+0000 7f32f2400640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32ec106510 0x7f32ec1a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.654+0000 7f32f2400640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32ec105b50 0x7f32ec19b0a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.654+0000 7f32f2400640 1 --2- 172.21.15.94:0/260099894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32ec104960 0x7f32ec19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.654+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 >> 172.21.15.94:0/260099894 conn(0x7f32ec0ffeb0 msgr2=0x7f32ec108f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.654+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 shutdown_connections 2024-04-26T20:12:22.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.654+0000 7f32f2400640 1 -- 172.21.15.94:0/260099894 wait complete. 2024-04-26T20:12:22.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:12:22.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.877+0000 7f38bf000640 1 -- 172.21.15.94:0/1970546029 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b8101060 msgr2=0x7f38b8101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.877+0000 7f38bf000640 1 --2- 172.21.15.94:0/1970546029 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b8101060 0x7f38b8101480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f38a8009ca0 tx=0x7f38a802a950 comp rx=0 tx=0).stop 2024-04-26T20:12:22.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.877+0000 7f38bf000640 1 -- 172.21.15.94:0/1970546029 shutdown_connections 2024-04-26T20:12:22.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.877+0000 7f38bf000640 1 --2- 172.21.15.94:0/1970546029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b8102360 0x7f38b81027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.877+0000 7f38bf000640 1 --2- 172.21.15.94:0/1970546029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38b81019c0 0x7f38b8101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.877+0000 7f38bf000640 1 --2- 172.21.15.94:0/1970546029 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b8101060 0x7f38b8101480 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.877+0000 7f38bf000640 1 -- 172.21.15.94:0/1970546029 >> 172.21.15.94:0/1970546029 conn(0x7f38b80fcbc0 msgr2=0x7f38b80ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:22.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.877+0000 7f38bf000640 1 -- 172.21.15.94:0/1970546029 shutdown_connections 2024-04-26T20:12:22.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.877+0000 7f38bf000640 1 -- 172.21.15.94:0/1970546029 wait complete. 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.878+0000 7f38bf000640 1 Processor -- start 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.878+0000 7f38bf000640 1 -- start start 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.878+0000 7f38bf000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b8101060 0x7f38b819ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.878+0000 7f38bf000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b81019c0 0x7f38b819b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.878+0000 7f38bf000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38b8102360 0x7f38b81a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.878+0000 7f38bf000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f38b8110470 con 0x7f38b8102360 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.878+0000 7f38bf000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38b81102f0 con 0x7f38b8101060 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.878+0000 7f38bf000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f38b81105f0 con 0x7f38b81019c0 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38be600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38b8102360 0x7f38b81a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bd200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b81019c0 0x7f38b819b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bdc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b8101060 0x7f38b819ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:22.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bd200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b81019c0 0x7f38b819b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:50482/0 (socket says 172.21.15.94:50482) 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bd200640 1 -- 172.21.15.94:0/1736473446 learned_addr learned my addr 172.21.15.94:0/1736473446 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bd200640 1 -- 172.21.15.94:0/1736473446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b8101060 msgr2=0x7f38b819ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bd200640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b8101060 0x7f38b819ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bd200640 1 -- 172.21.15.94:0/1736473446 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38b8102360 msgr2=0x7f38b81a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bd200640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38b8102360 0x7f38b81a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bd200640 1 -- 172.21.15.94:0/1736473446 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b81a2760 con 0x7f38b81019c0 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.879+0000 7f38bd200640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b81019c0 0x7f38b819b080 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f38a400ea60 tx=0x7f38a400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.880+0000 7f38b6000640 1 -- 172.21.15.94:0/1736473446 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a400cde0 con 0x7f38b81019c0 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.880+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b81a2a50 con 0x7f38b81019c0 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.880+0000 7f38b6000640 1 -- 172.21.15.94:0/1736473446 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38a40035f0 con 0x7f38b81019c0 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.880+0000 7f38b6000640 1 -- 172.21.15.94:0/1736473446 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4010690 con 0x7f38b81019c0 2024-04-26T20:12:22.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.880+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b81a2ff0 con 0x7f38b81019c0 2024-04-26T20:12:22.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.881+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b810bdb0 con 0x7f38b81019c0 2024-04-26T20:12:22.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.884+0000 7f38b6000640 1 -- 172.21.15.94:0/1736473446 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f38a40037a0 con 0x7f38b81019c0 2024-04-26T20:12:22.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.885+0000 7f38b6000640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f389c0a3c40 0x7f389c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:22.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.885+0000 7f38b6000640 1 -- 172.21.15.94:0/1736473446 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f38a40d0c50 con 0x7f38b81019c0 2024-04-26T20:12:22.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.885+0000 7f38bdc00640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f389c0a3c40 0x7f389c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:22.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.886+0000 7f38bdc00640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f389c0a3c40 0x7f389c0a6100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f38a8000f80 tx=0x7f38a802f890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:22.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:22.890+0000 7f38b6000640 1 -- 172.21.15.94:0/1736473446 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f38a409d0a0 con 0x7f38b81019c0 2024-04-26T20:12:22.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:22 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/535996027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:12:22.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:22 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/260099894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:12:23.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:22 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/535996027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:12:23.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:22 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/260099894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:12:23.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.093+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38b802dcf0 con 0x7f38b81019c0 2024-04-26T20:12:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.095+0000 7f38b6000640 1 -- 172.21.15.94:0/1736473446 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24879 (secure 0 0 0) 0x7f38a409c9a0 con 0x7f38b81019c0 2024-04-26T20:12:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:12:23.097 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:10:13.755106+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:12:23.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f389c0a3c40 msgr2=0x7f389c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f389c0a3c40 0x7f389c0a6100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f38a8000f80 tx=0x7f38a802f890 comp rx=0 tx=0).stop 2024-04-26T20:12:23.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b81019c0 msgr2=0x7f38b819b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b81019c0 0x7f38b819b080 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f38a400ea60 tx=0x7f38a400ef20 comp rx=0 tx=0).stop 2024-04-26T20:12:23.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 shutdown_connections 2024-04-26T20:12:23.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f389c0a3c40 0x7f389c0a6100 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38b8102360 0x7f38b81a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b81019c0 0x7f38b819b080 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 --2- 172.21.15.94:0/1736473446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b8101060 0x7f38b819ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 >> 172.21.15.94:0/1736473446 conn(0x7f38b80fcbc0 msgr2=0x7f38b80fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:23.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 shutdown_connections 2024-04-26T20:12:23.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.098+0000 7f38bf000640 1 -- 172.21.15.94:0/1736473446 wait complete. 2024-04-26T20:12:23.119 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:08:13.904332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:08:15.670247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:08:23.682553+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:12:23.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:12:23.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:22 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/535996027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:12:23.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:22 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/260099894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:12:23.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.313+0000 7f52a1600640 1 -- 172.21.15.94:0/3658819841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c103a50 msgr2=0x7f529c103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.313+0000 7f52a1600640 1 --2- 172.21.15.94:0/3658819841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c103a50 0x7f529c103ed0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5284009900 tx=0x7f528402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.313+0000 7f52a1600640 1 -- 172.21.15.94:0/3658819841 shutdown_connections 2024-04-26T20:12:23.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.313+0000 7f52a1600640 1 --2- 172.21.15.94:0/3658819841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f529c104410 0x7f529c10aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.313+0000 7f52a1600640 1 --2- 172.21.15.94:0/3658819841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c103a50 0x7f529c103ed0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.313+0000 7f52a1600640 1 --2- 172.21.15.94:0/3658819841 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f529c102860 0x7f529c102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.313+0000 7f52a1600640 1 -- 172.21.15.94:0/3658819841 >> 172.21.15.94:0/3658819841 conn(0x7f529c0fddd0 msgr2=0x7f529c100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:23.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.313+0000 7f52a1600640 1 -- 172.21.15.94:0/3658819841 shutdown_connections 2024-04-26T20:12:23.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.313+0000 7f52a1600640 1 -- 172.21.15.94:0/3658819841 wait complete. 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.314+0000 7f52a1600640 1 Processor -- start 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.314+0000 7f52a1600640 1 -- start start 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.314+0000 7f52a1600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f529c102860 0x7f529c19acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.314+0000 7f52a1600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f529c103a50 0x7f529c19b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.314+0000 7f52a1600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c104410 0x7f529c1a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.314+0000 7f52a1600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f529c10e3e0 con 0x7f529c103a50 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.314+0000 7f52a1600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f529c10e260 con 0x7f529c104410 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.314+0000 7f52a1600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f529c10e560 con 0x7f529c102860 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f52a0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c104410 0x7f529c1a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f5293400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f529c103a50 0x7f529c19b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f529be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f529c102860 0x7f529c19acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:23.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f52a0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c104410 0x7f529c1a2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:41622/0 (socket says 172.21.15.94:41622) 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f52a0c00640 1 -- 172.21.15.94:0/1696434684 learned_addr learned my addr 172.21.15.94:0/1696434684 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f52a0c00640 1 -- 172.21.15.94:0/1696434684 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f529c102860 msgr2=0x7f529c19acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f52a0c00640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f529c102860 0x7f529c19acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f52a0c00640 1 -- 172.21.15.94:0/1696434684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f529c103a50 msgr2=0x7f529c19b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f52a0c00640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f529c103a50 0x7f529c19b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f52a0c00640 1 -- 172.21.15.94:0/1696434684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529c1a28e0 con 0x7f529c104410 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.315+0000 7f52a0c00640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c104410 0x7f529c1a2240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f529400e970 tx=0x7f529400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.316+0000 7f5299600640 1 -- 172.21.15.94:0/1696434684 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529400ccc0 con 0x7f529c104410 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.316+0000 7f5299600640 1 -- 172.21.15.94:0/1696434684 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f529400ce60 con 0x7f529c104410 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.316+0000 7f5299600640 1 -- 172.21.15.94:0/1696434684 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5294010430 con 0x7f529c104410 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.316+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529c1a2bd0 con 0x7f529c104410 2024-04-26T20:12:23.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.316+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529c1a30a0 con 0x7f529c104410 2024-04-26T20:12:23.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.317+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f529c1a32d0 con 0x7f529c104410 2024-04-26T20:12:23.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.319+0000 7f5299600640 1 -- 172.21.15.94:0/1696434684 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5294004fb0 con 0x7f529c104410 2024-04-26T20:12:23.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.320+0000 7f5299600640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52800a3cc0 0x7f52800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:23.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.321+0000 7f5299600640 1 -- 172.21.15.94:0/1696434684 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f52940d0ee0 con 0x7f529c104410 2024-04-26T20:12:23.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.321+0000 7f529be00640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52800a3cc0 0x7f52800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:23.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.321+0000 7f529be00640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52800a3cc0 0x7f52800a6180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f529c100230 tx=0x7f528c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:23.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.322+0000 7f5299600640 1 -- 172.21.15.94:0/1696434684 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5294054060 con 0x7f529c104410 2024-04-26T20:12:23.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.522+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f529c1a34a0 con 0x7f529c104410 2024-04-26T20:12:23.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.523+0000 7f5299600640 1 -- 172.21.15.94:0/1696434684 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f529409d330 con 0x7f529c104410 2024-04-26T20:12:23.524 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:12:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52800a3cc0 msgr2=0x7f52800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52800a3cc0 0x7f52800a6180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f529c100230 tx=0x7f528c009500 comp rx=0 tx=0).stop 2024-04-26T20:12:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c104410 msgr2=0x7f529c1a2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c104410 0x7f529c1a2240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f529400e970 tx=0x7f529400ed70 comp rx=0 tx=0).stop 2024-04-26T20:12:23.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 shutdown_connections 2024-04-26T20:12:23.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f52800a3cc0 0x7f52800a6180 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f529c104410 0x7f529c1a2240 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f529c103a50 0x7f529c19b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 --2- 172.21.15.94:0/1696434684 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f529c102860 0x7f529c19acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 >> 172.21.15.94:0/1696434684 conn(0x7f529c0fddd0 msgr2=0x7f529c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:23.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 shutdown_connections 2024-04-26T20:12:23.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.526+0000 7f52a1600640 1 -- 172.21.15.94:0/1696434684 wait complete. 2024-04-26T20:12:23.546 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.753+0000 7f6e76a00640 1 -- 172.21.15.94:0/2370777897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e7010a980 msgr2=0x7f6e701112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.753+0000 7f6e76a00640 1 --2- 172.21.15.94:0/2370777897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e7010a980 0x7f6e701112a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009ca0 tx=0x7f6e6802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.753+0000 7f6e76a00640 1 -- 172.21.15.94:0/2370777897 shutdown_connections 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.753+0000 7f6e76a00640 1 --2- 172.21.15.94:0/2370777897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e7010a980 0x7f6e701112a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.753+0000 7f6e76a00640 1 --2- 172.21.15.94:0/2370777897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e70109fc0 0x7f6e7010a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.753+0000 7f6e76a00640 1 --2- 172.21.15.94:0/2370777897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e70108e70 0x7f6e70109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.753+0000 7f6e76a00640 1 -- 172.21.15.94:0/2370777897 >> 172.21.15.94:0/2370777897 conn(0x7f6e70077e00 msgr2=0x7f6e7007a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.753+0000 7f6e76a00640 1 -- 172.21.15.94:0/2370777897 shutdown_connections 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.753+0000 7f6e76a00640 1 -- 172.21.15.94:0/2370777897 wait complete. 2024-04-26T20:12:23.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.754+0000 7f6e76a00640 1 Processor -- start 2024-04-26T20:12:23.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.754+0000 7f6e76a00640 1 -- start start 2024-04-26T20:12:23.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.754+0000 7f6e76a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e70108e70 0x7f6e7019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:23.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.754+0000 7f6e76a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e70109fc0 0x7f6e7019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:23.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.754+0000 7f6e76a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e7010a980 0x7f6e701a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:23.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.754+0000 7f6e76a00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70114a20 con 0x7f6e70108e70 2024-04-26T20:12:23.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.754+0000 7f6e76a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701148a0 con 0x7f6e70109fc0 2024-04-26T20:12:23.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.754+0000 7f6e76a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70114ba0 con 0x7f6e7010a980 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e75600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e70108e70 0x7f6e7019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e74c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e70109fc0 0x7f6e7019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e74c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e70109fc0 0x7f6e7019f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:41630/0 (socket says 172.21.15.94:41630) 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e74c00640 1 -- 172.21.15.94:0/3111129006 learned_addr learned my addr 172.21.15.94:0/3111129006 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e76000640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e7010a980 0x7f6e701a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e74c00640 1 -- 172.21.15.94:0/3111129006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e7010a980 msgr2=0x7f6e701a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e74c00640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e7010a980 0x7f6e701a6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e74c00640 1 -- 172.21.15.94:0/3111129006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e70108e70 msgr2=0x7f6e7019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e74c00640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e70108e70 0x7f6e7019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e74c00640 1 -- 172.21.15.94:0/3111129006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e701a6c00 con 0x7f6e70109fc0 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.755+0000 7f6e74c00640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e70109fc0 0x7f6e7019f520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00e9c0 tx=0x7f6e5c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.756+0000 7f6e6e000640 1 -- 172.21.15.94:0/3111129006 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c00cd80 con 0x7f6e70109fc0 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.756+0000 7f6e6e000640 1 -- 172.21.15.94:0/3111129006 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e5c0035f0 con 0x7f6e70109fc0 2024-04-26T20:12:23.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.756+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e701a6ef0 con 0x7f6e70109fc0 2024-04-26T20:12:23.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.756+0000 7f6e6e000640 1 -- 172.21.15.94:0/3111129006 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c010690 con 0x7f6e70109fc0 2024-04-26T20:12:23.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.756+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e701a7360 con 0x7f6e70109fc0 2024-04-26T20:12:23.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.757+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e701100b0 con 0x7f6e70109fc0 2024-04-26T20:12:23.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.759+0000 7f6e6e000640 1 -- 172.21.15.94:0/3111129006 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6e5c004000 con 0x7f6e70109fc0 2024-04-26T20:12:23.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.761+0000 7f6e6e000640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6e580a3c40 0x7f6e580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:23.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.761+0000 7f6e6e000640 1 -- 172.21.15.94:0/3111129006 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f6e5c0d0f60 con 0x7f6e70109fc0 2024-04-26T20:12:23.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.761+0000 7f6e75600640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6e580a3c40 0x7f6e580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:23.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.762+0000 7f6e75600640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6e580a3c40 0x7f6e580a6100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009bd0 tx=0x7f6e64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:23.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.765+0000 7f6e6e000640 1 -- 172.21.15.94:0/3111129006 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6e5c09d3b0 con 0x7f6e70109fc0 2024-04-26T20:12:23.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.969+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6e7002dca0 con 0x7f6e70109fc0 2024-04-26T20:12:23.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.970+0000 7f6e6e000640 1 -- 172.21.15.94:0/3111129006 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f6e5c09ccb0 con 0x7f6e70109fc0 2024-04-26T20:12:23.971 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6e580a3c40 msgr2=0x7f6e580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6e580a3c40 0x7f6e580a6100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009bd0 tx=0x7f6e64005f90 comp rx=0 tx=0).stop 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e70109fc0 msgr2=0x7f6e7019f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e70109fc0 0x7f6e7019f520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00e9c0 tx=0x7f6e5c00ee80 comp rx=0 tx=0).stop 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 shutdown_connections 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f6e580a3c40 0x7f6e580a6100 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e7010a980 0x7f6e701a6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e70109fc0 0x7f6e7019f520 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 --2- 172.21.15.94:0/3111129006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e70108e70 0x7f6e7019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:23.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.972+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 >> 172.21.15.94:0/3111129006 conn(0x7f6e70077e00 msgr2=0x7f6e7010d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:23.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.973+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 shutdown_connections 2024-04-26T20:12:23.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:23.973+0000 7f6e76a00640 1 -- 172.21.15.94:0/3111129006 wait complete. 2024-04-26T20:12:23.986 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1736473446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:12:23.986 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:23 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1696434684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:12:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:23 smithi094 ceph-mon[28050]: pgmap v452: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:23.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:12:24.047 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1736473446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:12:24.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:23 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1696434684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:12:24.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:23 smithi204 ceph-mon[32618]: pgmap v452: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:24.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.200+0000 7f4712200640 1 -- 172.21.15.94:0/4249782105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c109f80 msgr2=0x7f470c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.200+0000 7f4712200640 1 --2- 172.21.15.94:0/4249782105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c109f80 0x7f470c10a400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009900 tx=0x7f46f802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.200+0000 7f4712200640 1 -- 172.21.15.94:0/4249782105 shutdown_connections 2024-04-26T20:12:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.200+0000 7f4712200640 1 --2- 172.21.15.94:0/4249782105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f470c10a940 0x7f470c1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.200+0000 7f4712200640 1 --2- 172.21.15.94:0/4249782105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c109f80 0x7f470c10a400 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.200+0000 7f4712200640 1 --2- 172.21.15.94:0/4249782105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f470c108d90 0x7f470c1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.200+0000 7f4712200640 1 -- 172.21.15.94:0/4249782105 >> 172.21.15.94:0/4249782105 conn(0x7f470c077d30 msgr2=0x7f470c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.200+0000 7f4712200640 1 -- 172.21.15.94:0/4249782105 shutdown_connections 2024-04-26T20:12:24.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.200+0000 7f4712200640 1 -- 172.21.15.94:0/4249782105 wait complete. 2024-04-26T20:12:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.201+0000 7f4712200640 1 Processor -- start 2024-04-26T20:12:24.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.201+0000 7f4712200640 1 -- start start 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f4712200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f470c108d90 0x7f470c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f4712200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f470c109f80 0x7f470c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f4712200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c10a940 0x7f470c1a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f4712200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f470c1148f0 con 0x7f470c109f80 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f4712200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f470c114770 con 0x7f470c108d90 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f4712200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f470c114a70 con 0x7f470c10a940 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f470be00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f470c109f80 0x7f470c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f4711800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c10a940 0x7f470c1a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f470be00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f470c109f80 0x7f470c19f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46396/0 (socket says 172.21.15.94:46396) 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f4710e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f470c108d90 0x7f470c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f4711800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c10a940 0x7f470c1a6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:50546/0 (socket says 172.21.15.94:50546) 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.202+0000 7f470be00640 1 -- 172.21.15.94:0/1374277143 learned_addr learned my addr 172.21.15.94:0/1374277143 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4711800640 1 -- 172.21.15.94:0/1374277143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f470c108d90 msgr2=0x7f470c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4711800640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f470c108d90 0x7f470c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4711800640 1 -- 172.21.15.94:0/1374277143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f470c109f80 msgr2=0x7f470c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4711800640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f470c109f80 0x7f470c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4711800640 1 -- 172.21.15.94:0/1374277143 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c1a6bb0 con 0x7f470c10a940 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4711800640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c10a940 0x7f470c1a6540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f470400da10 tx=0x7f470400ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4709600640 1 -- 172.21.15.94:0/1374277143 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470400bd70 con 0x7f470c10a940 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4709600640 1 -- 172.21.15.94:0/1374277143 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4704003e20 con 0x7f470c10a940 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c1a6ea0 con 0x7f470c10a940 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4709600640 1 -- 172.21.15.94:0/1374277143 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47040106c0 con 0x7f470c10a940 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.203+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c1a7310 con 0x7f470c10a940 2024-04-26T20:12:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.204+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c1100b0 con 0x7f470c10a940 2024-04-26T20:12:24.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.206+0000 7f4709600640 1 -- 172.21.15.94:0/1374277143 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4704004270 con 0x7f470c10a940 2024-04-26T20:12:24.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.208+0000 7f4709600640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46f40a3c40 0x7f46f40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:24.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.208+0000 7f4709600640 1 -- 172.21.15.94:0/1374277143 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f47040d0d30 con 0x7f470c10a940 2024-04-26T20:12:24.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.208+0000 7f4710e00640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46f40a3c40 0x7f46f40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:24.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.209+0000 7f4710e00640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46f40a3c40 0x7f46f40a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f470c109de0 tx=0x7f470000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:24.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.212+0000 7f4709600640 1 -- 172.21.15.94:0/1374277143 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f470409d180 con 0x7f470c10a940 2024-04-26T20:12:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1736473446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:12:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:23 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1696434684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:12:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:23 smithi103 ceph-mon[32635]: pgmap v452: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:24.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.412+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f470c02dcd0 con 0x7f470c10a940 2024-04-26T20:12:24.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.413+0000 7f4709600640 1 -- 172.21.15.94:0/1374277143 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f47040bd0c0 con 0x7f470c10a940 2024-04-26T20:12:24.414 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-26T20:12:24.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46f40a3c40 msgr2=0x7f46f40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46f40a3c40 0x7f46f40a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f470c109de0 tx=0x7f470000b390 comp rx=0 tx=0).stop 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c10a940 msgr2=0x7f470c1a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c10a940 0x7f470c1a6540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f470400da10 tx=0x7f470400ded0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 shutdown_connections 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46f40a3c40 0x7f46f40a6100 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f470c10a940 0x7f470c1a6540 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f470c109f80 0x7f470c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 --2- 172.21.15.94:0/1374277143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f470c108d90 0x7f470c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 >> 172.21.15.94:0/1374277143 conn(0x7f470c077d30 msgr2=0x7f470c10d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:24.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 shutdown_connections 2024-04-26T20:12:24.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.416+0000 7f4712200640 1 -- 172.21.15.94:0/1374277143 wait complete. 2024-04-26T20:12:24.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:12:24.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.639+0000 7f9b12800640 1 -- 172.21.15.94:0/2596793902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 msgr2=0x7f9b0c101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.639+0000 7f9b12800640 1 --2- 172.21.15.94:0/2596793902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 0x7f9b0c101ee0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009900 tx=0x7f9af802a400 comp rx=0 tx=0).stop 2024-04-26T20:12:24.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.639+0000 7f9b12800640 1 -- 172.21.15.94:0/2596793902 shutdown_connections 2024-04-26T20:12:24.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.639+0000 7f9b12800640 1 --2- 172.21.15.94:0/2596793902 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b0c102420 0x7f9b0c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.639+0000 7f9b12800640 1 --2- 172.21.15.94:0/2596793902 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 0x7f9b0c101ee0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.639+0000 7f9b12800640 1 --2- 172.21.15.94:0/2596793902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b0c101120 0x7f9b0c101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.639+0000 7f9b12800640 1 -- 172.21.15.94:0/2596793902 >> 172.21.15.94:0/2596793902 conn(0x7f9b0c0fcce0 msgr2=0x7f9b0c0ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:24.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.639+0000 7f9b12800640 1 -- 172.21.15.94:0/2596793902 shutdown_connections 2024-04-26T20:12:24.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.639+0000 7f9b12800640 1 -- 172.21.15.94:0/2596793902 wait complete. 2024-04-26T20:12:24.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.640+0000 7f9b12800640 1 Processor -- start 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.640+0000 7f9b12800640 1 -- start start 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.640+0000 7f9b12800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b0c101120 0x7f9b0c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b12800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 0x7f9b0c19b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b12800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b0c102420 0x7f9b0c1a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b12800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c110570 con 0x7f9b0c101a80 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b12800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c1103f0 con 0x7f9b0c102420 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b12800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c1106f0 con 0x7f9b0c101120 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b11400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b0c101120 0x7f9b0c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b10a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 0x7f9b0c19b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b11400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b0c101120 0x7f9b0c19ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:50568/0 (socket says 172.21.15.94:50568) 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b10a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 0x7f9b0c19b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46416/0 (socket says 172.21.15.94:46416) 2024-04-26T20:12:24.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b11400640 1 -- 172.21.15.94:0/1563350008 learned_addr learned my addr 172.21.15.94:0/1563350008 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b11e00640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b0c102420 0x7f9b0c1a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b11400640 1 -- 172.21.15.94:0/1563350008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b0c102420 msgr2=0x7f9b0c1a2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b11400640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b0c102420 0x7f9b0c1a2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b11400640 1 -- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 msgr2=0x7f9b0c19b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b11400640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 0x7f9b0c19b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b11400640 1 -- 172.21.15.94:0/1563350008 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0c1a28b0 con 0x7f9b0c101120 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.641+0000 7f9b10a00640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 0x7f9b0c19b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.642+0000 7f9b11400640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b0c101120 0x7f9b0c19ac70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000e9f0 tx=0x7f9b0000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.642+0000 7f9b0a000640 1 -- 172.21.15.94:0/1563350008 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0000cd80 con 0x7f9b0c101120 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.642+0000 7f9b0a000640 1 -- 172.21.15.94:0/1563350008 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b000108e0 con 0x7f9b0c101120 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.642+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0c1a2ba0 con 0x7f9b0c101120 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.642+0000 7f9b0a000640 1 -- 172.21.15.94:0/1563350008 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00018980 con 0x7f9b0c101120 2024-04-26T20:12:24.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.642+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b0c1a30c0 con 0x7f9b0c101120 2024-04-26T20:12:24.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.643+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b0c10c040 con 0x7f9b0c101120 2024-04-26T20:12:24.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.645+0000 7f9b0a000640 1 -- 172.21.15.94:0/1563350008 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9b00010430 con 0x7f9b0c101120 2024-04-26T20:12:24.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.646+0000 7f9b0a000640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9af40a3c40 0x7f9af40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:24.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.646+0000 7f9b0a000640 1 -- 172.21.15.94:0/1563350008 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f9b000d0d20 con 0x7f9b0c101120 2024-04-26T20:12:24.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.646+0000 7f9b10a00640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9af40a3c40 0x7f9af40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:24.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.647+0000 7f9b10a00640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9af40a3c40 0x7f9af40a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9af8000c00 tx=0x7f9af80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:24.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.650+0000 7f9b0a000640 1 -- 172.21.15.94:0/1563350008 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9b0009d170 con 0x7f9b0c101120 2024-04-26T20:12:24.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.857+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9b0c063530 con 0x7f9b0c101120 2024-04-26T20:12:24.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.858+0000 7f9b0a000640 1 -- 172.21.15.94:0/1563350008 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f9b0009ca70 con 0x7f9b0c101120 2024-04-26T20:12:24.859 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:12:24.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9af40a3c40 msgr2=0x7f9af40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9af40a3c40 0x7f9af40a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9af8000c00 tx=0x7f9af80314a0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b0c101120 msgr2=0x7f9b0c19ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:24.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b0c101120 0x7f9b0c19ac70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000e9f0 tx=0x7f9b0000eeb0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 shutdown_connections 2024-04-26T20:12:24.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f9af40a3c40 0x7f9af40a6100 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b0c102420 0x7f9b0c1a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b0c101a80 0x7f9b0c19b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 --2- 172.21.15.94:0/1563350008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b0c101120 0x7f9b0c19ac70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:24.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 >> 172.21.15.94:0/1563350008 conn(0x7f9b0c0fcce0 msgr2=0x7f9b0c0fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:24.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 shutdown_connections 2024-04-26T20:12:24.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:24.861+0000 7f9b12800640 1 -- 172.21.15.94:0/1563350008 wait complete. 2024-04-26T20:12:24.881 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:12:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.084+0000 7f4678400640 1 -- 172.21.15.94:0/985614288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670106740 msgr2=0x7f467010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.084+0000 7f4678400640 1 --2- 172.21.15.94:0/985614288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670106740 0x7f467010cfd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4668009ca0 tx=0x7f466802a8b0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.084+0000 7f4678400640 1 -- 172.21.15.94:0/985614288 shutdown_connections 2024-04-26T20:12:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.084+0000 7f4678400640 1 --2- 172.21.15.94:0/985614288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670106740 0x7f467010cfd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.084+0000 7f4678400640 1 --2- 172.21.15.94:0/985614288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4670105d80 0x7f4670106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.084+0000 7f4678400640 1 --2- 172.21.15.94:0/985614288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4670104b90 0x7f4670104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.084+0000 7f4678400640 1 -- 172.21.15.94:0/985614288 >> 172.21.15.94:0/985614288 conn(0x7f4670100120 msgr2=0x7f4670102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:25.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.084+0000 7f4678400640 1 -- 172.21.15.94:0/985614288 shutdown_connections 2024-04-26T20:12:25.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.084+0000 7f4678400640 1 -- 172.21.15.94:0/985614288 wait complete. 2024-04-26T20:12:25.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.085+0000 7f4678400640 1 Processor -- start 2024-04-26T20:12:25.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.085+0000 7f4678400640 1 -- start start 2024-04-26T20:12:25.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.085+0000 7f4678400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670104b90 0x7f467019ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:25.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.085+0000 7f4678400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4670105d80 0x7f467019b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:25.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.085+0000 7f4678400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4670106740 0x7f46701a2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:25.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.085+0000 7f4678400640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4670110710 con 0x7f4670105d80 2024-04-26T20:12:25.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.085+0000 7f4678400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4670110590 con 0x7f4670106740 2024-04-26T20:12:25.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.085+0000 7f4678400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4670110890 con 0x7f4670104b90 2024-04-26T20:12:25.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670104b90 0x7f467019ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:25.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670104b90 0x7f467019ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:50588/0 (socket says 172.21.15.94:50588) 2024-04-26T20:12:25.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4670105d80 0x7f467019b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:25.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675a00640 1 -- 172.21.15.94:0/2288774682 learned_addr learned my addr 172.21.15.94:0/2288774682 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:25.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4676400640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4670106740 0x7f46701a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:25.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675a00640 1 -- 172.21.15.94:0/2288774682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4670106740 msgr2=0x7f46701a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675a00640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4670106740 0x7f46701a2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675a00640 1 -- 172.21.15.94:0/2288774682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4670105d80 msgr2=0x7f467019b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675a00640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4670105d80 0x7f467019b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675a00640 1 -- 172.21.15.94:0/2288774682 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46701a2930 con 0x7f4670104b90 2024-04-26T20:12:25.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.086+0000 7f4675a00640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670104b90 0x7f467019ad20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f466400e9c0 tx=0x7f466400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:25.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.087+0000 7f466e000640 1 -- 172.21.15.94:0/2288774682 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466400cd80 con 0x7f4670104b90 2024-04-26T20:12:25.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.087+0000 7f466e000640 1 -- 172.21.15.94:0/2288774682 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46640035f0 con 0x7f4670104b90 2024-04-26T20:12:25.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.087+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46701a2c20 con 0x7f4670104b90 2024-04-26T20:12:25.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.087+0000 7f466e000640 1 -- 172.21.15.94:0/2288774682 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4664010690 con 0x7f4670104b90 2024-04-26T20:12:25.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.087+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46701a3170 con 0x7f4670104b90 2024-04-26T20:12:25.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.088+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f467010bf30 con 0x7f4670104b90 2024-04-26T20:12:25.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.090+0000 7f466e000640 1 -- 172.21.15.94:0/2288774682 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4664004000 con 0x7f4670104b90 2024-04-26T20:12:25.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.091+0000 7f466e000640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46580a3cc0 0x7f46580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:25.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.091+0000 7f466e000640 1 -- 172.21.15.94:0/2288774682 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f46640d0eb0 con 0x7f4670104b90 2024-04-26T20:12:25.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.091+0000 7f4675000640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46580a3cc0 0x7f46580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:25.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.092+0000 7f4675000640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46580a3cc0 0x7f46580a6180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f465c0095f0 tx=0x7f465c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:25.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.095+0000 7f466e000640 1 -- 172.21.15.94:0/2288774682 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f466409d300 con 0x7f4670104b90 2024-04-26T20:12:25.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:24 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3111129006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:12:25.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:24 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1374277143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:12:25.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:24 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3111129006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:12:25.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:24 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1374277143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:12:25.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:24 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3111129006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:12:25.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:24 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1374277143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:12:25.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.340+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f467002dad0 con 0x7f4670104b90 2024-04-26T20:12:25.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.341+0000 7f466e000640 1 -- 172.21.15.94:0/2288774682 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11878 (secure 0 0 0) 0x7f466409cc00 con 0x7f4670104b90 2024-04-26T20:12:25.343 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:12:25.343 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:09:27.103941+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24562,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24562,24541]},"id":1}]} 2024-04-26T20:12:25.343 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 24 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.344+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46580a3cc0 msgr2=0x7f46580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.344+0000 7f4678400640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46580a3cc0 0x7f46580a6180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f465c0095f0 tx=0x7f465c005f90 comp rx=0 tx=0).stop 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.344+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670104b90 msgr2=0x7f467019ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.344+0000 7f4678400640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670104b90 0x7f467019ad20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f466400e9c0 tx=0x7f466400ee80 comp rx=0 tx=0).stop 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.345+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 shutdown_connections 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.345+0000 7f4678400640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f46580a3cc0 0x7f46580a6180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.345+0000 7f4678400640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4670106740 0x7f46701a2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.345+0000 7f4678400640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4670105d80 0x7f467019b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.345+0000 7f4678400640 1 --2- 172.21.15.94:0/2288774682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4670104b90 0x7f467019ad20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.345+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 >> 172.21.15.94:0/2288774682 conn(0x7f4670100120 msgr2=0x7f4670109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:25.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.345+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 shutdown_connections 2024-04-26T20:12:25.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.345+0000 7f4678400640 1 -- 172.21.15.94:0/2288774682 wait complete. 2024-04-26T20:12:25.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:12:25.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.572+0000 7ff72c200640 1 -- 172.21.15.94:0/2761757543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff72410a180 msgr2=0x7ff72410a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.572+0000 7ff72c200640 1 --2- 172.21.15.94:0/2761757543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff72410a180 0x7ff72410a600 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff710009860 tx=0x7ff71002a340 comp rx=0 tx=0).stop 2024-04-26T20:12:25.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.572+0000 7ff72c200640 1 -- 172.21.15.94:0/2761757543 shutdown_connections 2024-04-26T20:12:25.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.572+0000 7ff72c200640 1 --2- 172.21.15.94:0/2761757543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff72410ab40 0x7ff7241113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.572+0000 7ff72c200640 1 --2- 172.21.15.94:0/2761757543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff72410a180 0x7ff72410a600 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.572+0000 7ff72c200640 1 --2- 172.21.15.94:0/2761757543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff724108f90 0x7ff7241093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.572+0000 7ff72c200640 1 -- 172.21.15.94:0/2761757543 >> 172.21.15.94:0/2761757543 conn(0x7ff724077f40 msgr2=0x7ff72407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:25.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.572+0000 7ff72c200640 1 -- 172.21.15.94:0/2761757543 shutdown_connections 2024-04-26T20:12:25.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.572+0000 7ff72c200640 1 -- 172.21.15.94:0/2761757543 wait complete. 2024-04-26T20:12:25.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.573+0000 7ff72c200640 1 Processor -- start 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.573+0000 7ff72c200640 1 -- start start 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff72c200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff724108f90 0x7ff72419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff72c200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff72410a180 0x7ff72419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff72c200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff72410ab40 0x7ff7241a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff72c200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff724114ad0 con 0x7ff724108f90 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff72c200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff724114950 con 0x7ff72410ab40 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff72c200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff724114c50 con 0x7ff72410a180 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff729800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff724108f90 0x7ff72419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff729800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff724108f90 0x7ff72419f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46460/0 (socket says 172.21.15.94:46460) 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff729800640 1 -- 172.21.15.94:0/3943343417 learned_addr learned my addr 172.21.15.94:0/3943343417 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff72a200640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff72410ab40 0x7ff7241a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff728e00640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff72410a180 0x7ff72419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:25.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff729800640 1 -- 172.21.15.94:0/3943343417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff72410a180 msgr2=0x7ff72419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff729800640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff72410a180 0x7ff72419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff729800640 1 -- 172.21.15.94:0/3943343417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff72410ab40 msgr2=0x7ff7241a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.574+0000 7ff729800640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff72410ab40 0x7ff7241a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.575+0000 7ff729800640 1 -- 172.21.15.94:0/3943343417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7241a6de0 con 0x7ff724108f90 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.575+0000 7ff72a200640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff72410ab40 0x7ff7241a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.575+0000 7ff729800640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff724108f90 0x7ff72419f1c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff71800e9f0 tx=0x7ff71800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.575+0000 7ff722000640 1 -- 172.21.15.94:0/3943343417 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71800ccc0 con 0x7ff724108f90 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.575+0000 7ff722000640 1 -- 172.21.15.94:0/3943343417 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7180035f0 con 0x7ff724108f90 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.575+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7241a70d0 con 0x7ff724108f90 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.575+0000 7ff722000640 1 -- 172.21.15.94:0/3943343417 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71800ce60 con 0x7ff724108f90 2024-04-26T20:12:25.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.575+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7241a75f0 con 0x7ff724108f90 2024-04-26T20:12:25.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.576+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7241102e0 con 0x7ff724108f90 2024-04-26T20:12:25.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.578+0000 7ff722000640 1 -- 172.21.15.94:0/3943343417 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff718004000 con 0x7ff724108f90 2024-04-26T20:12:25.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.580+0000 7ff722000640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff70c0a3c40 0x7ff70c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:25.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.580+0000 7ff722000640 1 -- 172.21.15.94:0/3943343417 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7ff7180d0b90 con 0x7ff724108f90 2024-04-26T20:12:25.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.580+0000 7ff728e00640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff70c0a3c40 0x7ff70c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:25.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.581+0000 7ff728e00640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff70c0a3c40 0x7ff70c0a6100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff71002a930 tx=0x7ff710030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.584+0000 7ff722000640 1 -- 172.21.15.94:0/3943343417 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff71809cfe0 con 0x7ff724108f90 2024-04-26T20:12:25.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.801+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff72402dad0 con 0x7ff724108f90 2024-04-26T20:12:25.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.806+0000 7ff722000640 1 -- 172.21.15.94:0/3943343417 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24879 (secure 0 0 0) 0x7ff71809c8e0 con 0x7ff724108f90 2024-04-26T20:12:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:12:25.808 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:10:13.755106+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:12:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.808+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff70c0a3c40 msgr2=0x7ff70c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.808+0000 7ff72c200640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff70c0a3c40 0x7ff70c0a6100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff71002a930 tx=0x7ff710030450 comp rx=0 tx=0).stop 2024-04-26T20:12:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff724108f90 msgr2=0x7ff72419f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff724108f90 0x7ff72419f1c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff71800e9f0 tx=0x7ff71800edf0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 shutdown_connections 2024-04-26T20:12:25.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff70c0a3c40 0x7ff70c0a6100 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff72410ab40 0x7ff7241a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff72410a180 0x7ff72419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 --2- 172.21.15.94:0/3943343417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff724108f90 0x7ff72419f1c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:25.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 >> 172.21.15.94:0/3943343417 conn(0x7ff724077f40 msgr2=0x7ff72410d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:25.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 shutdown_connections 2024-04-26T20:12:25.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:25.809+0000 7ff72c200640 1 -- 172.21.15.94:0/3943343417 wait complete. 2024-04-26T20:12:25.831 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T20:08:18.399846+0000', 'modified': '2024-04-26T20:09:27.103941+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24575, 'mds_1': 24562, 'mds_2': 24541}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6835/4163398350', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6834', 'nonce': 4163398350}, {'type': 'v1', 'addr': '172.21.15.204:6835', 'nonce': 4163398350}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6837/820565196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6836', 'nonce': 820565196}, {'type': 'v1', 'addr': '172.21.15.204:6837', 'nonce': 820565196}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6837/3697356689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 3697356689}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 3697356689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.204:6839/3633033106', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6838', 'nonce': 3633033106}, {'type': 'v1', 'addr': '172.21.15.204:6839', 'nonce': 3633033106}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24562': {'gid': 24562, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 22, 'addr': '172.21.15.94:6841/989331046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6840', 'nonce': 989331046}, {'type': 'v1', 'addr': '172.21.15.94:6841', 'nonce': 989331046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.103:6839/1048742250', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 1048742250}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 1048742250}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24575, 'qdb_cluster': [24575, 24562, 24541]} 2024-04-26T20:12:25.832 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T20:12:25.832 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-26T20:12:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.034+0000 7f78b1600640 1 -- 172.21.15.94:0/4161587688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10ab60 msgr2=0x7f78ac1113f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.034+0000 7f78b1600640 1 --2- 172.21.15.94:0/4161587688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10ab60 0x7f78ac1113f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009ca0 tx=0x7f78a402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.034+0000 7f78b1600640 1 -- 172.21.15.94:0/4161587688 shutdown_connections 2024-04-26T20:12:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.034+0000 7f78b1600640 1 --2- 172.21.15.94:0/4161587688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10ab60 0x7f78ac1113f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.034+0000 7f78b1600640 1 --2- 172.21.15.94:0/4161587688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78ac10a1a0 0x7f78ac10a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.034+0000 7f78b1600640 1 --2- 172.21.15.94:0/4161587688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78ac108fb0 0x7f78ac1093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.034+0000 7f78b1600640 1 -- 172.21.15.94:0/4161587688 >> 172.21.15.94:0/4161587688 conn(0x7f78ac077f40 msgr2=0x7f78ac07a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.035+0000 7f78b1600640 1 -- 172.21.15.94:0/4161587688 shutdown_connections 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.035+0000 7f78b1600640 1 -- 172.21.15.94:0/4161587688 wait complete. 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78b1600640 1 Processor -- start 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78b1600640 1 -- start start 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78b1600640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78ac108fb0 0x7f78ac19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78b1600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10a1a0 0x7f78ac19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78b1600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78ac10ab60 0x7f78ac1a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78b1600640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac114b40 con 0x7f78ac108fb0 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78b1600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1149c0 con 0x7f78ac10ab60 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78b1600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac114cc0 con 0x7f78ac10a1a0 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78a2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10a1a0 0x7f78ac19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78aaa00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78ac108fb0 0x7f78ac19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78a2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10a1a0 0x7f78ac19f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:50624/0 (socket says 172.21.15.94:50624) 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78a2000640 1 -- 172.21.15.94:0/3890600695 learned_addr learned my addr 172.21.15.94:0/3890600695 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.036+0000 7f78ab400640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78ac10ab60 0x7f78ac1a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78a2000640 1 -- 172.21.15.94:0/3890600695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78ac10ab60 msgr2=0x7f78ac1a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78a2000640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78ac10ab60 0x7f78ac1a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78a2000640 1 -- 172.21.15.94:0/3890600695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78ac108fb0 msgr2=0x7f78ac19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78a2000640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78ac108fb0 0x7f78ac19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78a2000640 1 -- 172.21.15.94:0/3890600695 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac1a6d20 con 0x7f78ac10a1a0 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78ab400640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78ac10ab60 0x7f78ac1a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78a2000640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10a1a0 0x7f78ac19f640 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f789400e9f0 tx=0x7f789400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78a3e00640 1 -- 172.21.15.94:0/3890600695 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789400cd80 con 0x7f78ac10a1a0 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78a3e00640 1 -- 172.21.15.94:0/3890600695 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78940108e0 con 0x7f78ac10a1a0 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78a3e00640 1 -- 172.21.15.94:0/3890600695 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7894018980 con 0x7f78ac10a1a0 2024-04-26T20:12:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.037+0000 7f78b1600640 1 -- 172.21.15.94:0/3890600695 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac1a6fb0 con 0x7f78ac10a1a0 2024-04-26T20:12:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.038+0000 7f78b1600640 1 -- 172.21.15.94:0/3890600695 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac1a7480 con 0x7f78ac10a1a0 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.040+0000 7f78b43b2740 1 Processor -- start 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.040+0000 7f78a3e00640 1 -- 172.21.15.94:0/3890600695 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7894010430 con 0x7f78ac10a1a0 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.040+0000 7f78b43b2740 1 -- start start 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff40f0 0x56510cff4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 0x56510cff95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff9c50 0x56510cffc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78b43b2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56510cf58600 con 0x56510cff40f0 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78b43b2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56510cef5100 con 0x56510cff4ae0 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78b43b2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56510cf545c0 con 0x56510cff9c50 2024-04-26T20:12:26.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78ab400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff40f0 0x56510cff4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78ab400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff40f0 0x56510cff4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46490/0 (socket says 172.21.15.94:46490) 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78ab400640 1 -- 172.21.15.94:0/2535659370 learned_addr learned my addr 172.21.15.94:0/2535659370 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78aaa00640 1 --2- 172.21.15.94:0/2535659370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 0x56510cff95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78a2000640 1 --2- 172.21.15.94:0/2535659370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff9c50 0x56510cffc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.041+0000 7f78ab400640 1 -- 172.21.15.94:0/2535659370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff9c50 msgr2=0x56510cffc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.042+0000 7f78ab400640 1 --2- 172.21.15.94:0/2535659370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff9c50 0x56510cffc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.042+0000 7f78ab400640 1 -- 172.21.15.94:0/2535659370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 msgr2=0x56510cff95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.042+0000 7f78ab400640 1 --2- 172.21.15.94:0/2535659370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 0x56510cff95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.042+0000 7f78ab400640 1 -- 172.21.15.94:0/2535659370 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56510cc6a2d0 con 0x56510cff40f0 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.042+0000 7f78ab400640 1 --2- 172.21.15.94:0/2535659370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff40f0 0x56510cff4510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f78a4005e30 tx=0x7f78a4005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c9cea54915f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.042+0000 7f789b400640 1 -- 172.21.15.94:0/2535659370 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4031650 con 0x56510cff40f0 2024-04-26T20:12:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.042+0000 7f78a3e00640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78900a3cc0 0x7f78900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.042+0000 7f789b400640 1 -- 172.21.15.94:0/2535659370 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a4036040 con 0x56510cff40f0 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.042+0000 7f789b400640 1 -- 172.21.15.94:0/2535659370 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a40046b0 con 0x56510cff40f0 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78a3e00640 1 -- 172.21.15.94:0/3890600695 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f78940d0cd0 con 0x7f78ac10a1a0 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78aaa00640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78900a3cc0 0x7f78900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78b43b2740 1 -- 172.21.15.94:0/2535659370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff40f0 msgr2=0x56510cff4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78b43b2740 1 --2- 172.21.15.94:0/2535659370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff40f0 0x56510cff4510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f78a4005e30 tx=0x7f78a4005f70 comp rx=0 tx=0).stop 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78aaa00640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78900a3cc0 0x7f78900a6180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x56510cff4ee0 tx=0x7f789c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78b43b2740 1 -- 172.21.15.94:0/2535659370 shutdown_connections 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78b43b2740 1 --2- 172.21.15.94:0/2535659370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff9c50 0x56510cffc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78b43b2740 1 --2- 172.21.15.94:0/2535659370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 0x56510cff95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78b43b2740 1 --2- 172.21.15.94:0/2535659370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff40f0 0x56510cff4510 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78b43b2740 1 -- 172.21.15.94:0/2535659370 >> 172.21.15.94:0/2535659370 conn(0x56510cf88d10 msgr2=0x56510cfefb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78b43b2740 1 -- 172.21.15.94:0/2535659370 shutdown_connections 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.043+0000 7f78b43b2740 1 -- 172.21.15.94:0/2535659370 wait complete. 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.044+0000 7f78b43b2740 1 -- start start 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.044+0000 7f78b43b2740 1 Processor -- start 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.044+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff40f0 0x56510cffe4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.044+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff4ae0 0x56510cffea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.044+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.044+0000 7f78b43b2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56510cffcba0 con 0x56510cff9c50 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.044+0000 7f78b43b2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56510cedff70 con 0x56510cff40f0 2024-04-26T20:12:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.044+0000 7f78b43b2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56510cffcd20 con 0x56510cff4ae0 2024-04-26T20:12:26.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78b43b2740 10 client.0 _ll_register_callbacks cb 0x56510cf4e480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:12:26.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78ab400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78ab400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46500/0 (socket says 172.21.15.94:46500) 2024-04-26T20:12:26.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78ab400640 1 -- 172.21.15.94:0/1110493175 learned_addr learned my addr 172.21.15.94:0/1110493175 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78aaa00640 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff40f0 0x56510cffe4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78a2000640 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff4ae0 0x56510cffea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78ab400640 1 -- 172.21.15.94:0/1110493175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff4ae0 msgr2=0x56510cffea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78ab400640 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff4ae0 0x56510cffea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78ab400640 1 -- 172.21.15.94:0/1110493175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff40f0 msgr2=0x56510cffe4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78ab400640 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff40f0 0x56510cffe4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78ab400640 1 -- 172.21.15.94:0/1110493175 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56510cd346b0 con 0x56510cff9c50 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f78ab400640 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005a60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f78a402ade0 tx=0x7f78a402f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85fc2e0ac69f04b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.045+0000 7f788fe00640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.046+0000 7f788fe00640 1 -- 172.21.15.94:0/1110493175 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a40315d0 con 0x56510cff9c50 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.046+0000 7f788fe00640 1 -- 172.21.15.94:0/1110493175 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f78a4031770 con 0x56510cff9c50 2024-04-26T20:12:26.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.046+0000 7f78b43b2740 1 -- 172.21.15.94:0/1110493175 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78a4036040 con 0x56510cff9c50 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.046+0000 7f788fe00640 1 -- 172.21.15.94:0/1110493175 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f78a4004430 con 0x56510cff9c50 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.046+0000 7f78b43b2740 10 client.15219 fetch_fsmap learned FSMap version 24 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.046+0000 7f78b43b2740 1 -- 172.21.15.94:0/1110493175 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56510cffcba0 con 0x56510cff9c50 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.046+0000 7f788fe00640 1 -- 172.21.15.94:0/1110493175 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a4036040 con 0x56510cff9c50 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.046+0000 7f788fe00640 1 -- 172.21.15.94:0/1110493175 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f78a402f730 con 0x56510cff9c50 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.046+0000 7f78b43b2740 10 client.15219 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.047+0000 7f78b43b2740 10 client.15219 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.047+0000 7f78b43b2740 20 client.15219 populate_metadata read hostname 'smithi094' 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.047+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x56510d00ac50 0x56510d00c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.047+0000 7f78b43b2740 4 client.15219 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.047+0000 7f78b43b2740 1 -- 172.21.15.94:0/1110493175 --> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56510cf66210 con 0x56510d00ac50 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.047+0000 7f78aaa00640 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x56510d00ac50 0x56510d00c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.047+0000 7f78aaa00640 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x56510d00ac50 0x56510d00c0c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7bff95b4a17eacc2 server_cookie=11a38883c823218 in_seq=0 out_seq=0 2024-04-26T20:12:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.047+0000 7f788fe00640 10 client.15219 ms_handle_connect on v2:172.21.15.204:6838/3633033106 2024-04-26T20:12:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.048+0000 7f788fe00640 1 -- 172.21.15.94:0/1110493175 <== mds.2 v2:172.21.15.204:6838/3633033106 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56510cf66210 con 0x56510d00ac50 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.048+0000 7f788fe00640 10 client.15219 handle_command_reply: tid=0 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.048+0000 7f78b43b2740 1 client.15219 shutdown 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 -- 172.21.15.94:0/1110493175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 msgr2=0x56510d005a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005a60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f78a402ade0 tx=0x7f78a402f3c0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 -- 172.21.15.94:0/1110493175 shutdown_connections 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x56510d00ac50 0x56510d00c0c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005a60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff4ae0 0x56510cffea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1110493175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff40f0 0x56510cffe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f788fe00640 0 client.15219 ms_handle_reset on v2:172.21.15.204:6838/3633033106 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 -- 172.21.15.94:0/1110493175 >> 172.21.15.94:0/1110493175 conn(0x56510cf88d10 msgr2=0x56510cfeeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 -- 172.21.15.94:0/1110493175 shutdown_connections 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 1 -- 172.21.15.94:0/1110493175 wait complete. 2024-04-26T20:12:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.049+0000 7f78b43b2740 20 client.15219 trim_cache size 0 max 16384 2024-04-26T20:12:26.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.069+0000 7f78b43b2740 1 Processor -- start 2024-04-26T20:12:26.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.069+0000 7f78b43b2740 1 -- start start 2024-04-26T20:12:26.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.069+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510cffe4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.069+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff4ae0 0x56510d005a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.069+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff9c50 0x56510cffe9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.069+0000 7f78b43b2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56510d00a400 con 0x56510cff4ae0 2024-04-26T20:12:26.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.069+0000 7f78b43b2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56510d00aad0 con 0x56510cff9c50 2024-04-26T20:12:26.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.069+0000 7f78b43b2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56510d00b1a0 con 0x56510cff40f0 2024-04-26T20:12:26.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78aaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510cffe4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78a2000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff4ae0 0x56510d005a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78aaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510cffe4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:50656/0 (socket says 172.21.15.94:50656) 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78aaa00640 1 -- 172.21.15.94:0/1626994783 learned_addr learned my addr 172.21.15.94:0/1626994783 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78ab400640 1 --2- 172.21.15.94:0/1626994783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff9c50 0x56510cffe9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78aaa00640 1 -- 172.21.15.94:0/1626994783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff9c50 msgr2=0x56510cffe9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78aaa00640 1 --2- 172.21.15.94:0/1626994783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff9c50 0x56510cffe9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78aaa00640 1 -- 172.21.15.94:0/1626994783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff4ae0 msgr2=0x56510d005a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78aaa00640 1 --2- 172.21.15.94:0/1626994783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff4ae0 0x56510d005a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78aaa00640 1 -- 172.21.15.94:0/1626994783 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56510cfff380 con 0x56510cff40f0 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.070+0000 7f78aaa00640 1 --2- 172.21.15.94:0/1626994783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510cffe4a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x56510cff0a20 tx=0x7f789c01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=749ec704f5bed650 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f789aa00640 1 -- 172.21.15.94:0/1626994783 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c02b420 con 0x56510cff40f0 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f789aa00640 1 -- 172.21.15.94:0/1626994783 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f789c0039c0 con 0x56510cff40f0 2024-04-26T20:12:26.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f789aa00640 1 -- 172.21.15.94:0/1626994783 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c005300 con 0x56510cff40f0 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f78b43b2740 1 -- 172.21.15.94:0/1626994783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 msgr2=0x56510cffe4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1626994783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510cffe4a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x56510cff0a20 tx=0x7f789c01a000 comp rx=0 tx=0).stop 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f78b43b2740 1 -- 172.21.15.94:0/1626994783 shutdown_connections 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1626994783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff9c50 0x56510cffe9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1626994783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff4ae0 0x56510d005a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f78b43b2740 1 --2- 172.21.15.94:0/1626994783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510cffe4a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f78b43b2740 1 -- 172.21.15.94:0/1626994783 >> 172.21.15.94:0/1626994783 conn(0x56510cf88d10 msgr2=0x56510cff5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f78b43b2740 1 -- 172.21.15.94:0/1626994783 shutdown_connections 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.071+0000 7f78b43b2740 1 -- 172.21.15.94:0/1626994783 wait complete. 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.072+0000 7f78b43b2740 1 -- start start 2024-04-26T20:12:26.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.072+0000 7f78b43b2740 1 Processor -- start 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.072+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510d001bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 0x56510d0020f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78b43b2740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78b43b2740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56510d00bf40 con 0x56510cff9c50 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78b43b2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56510d00b870 con 0x56510cff4ae0 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78b43b2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56510d00c610 con 0x56510cff40f0 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78b43b2740 10 client.0 _ll_register_callbacks cb 0x7f789c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78aaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510d001bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78a2000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 0x56510d0020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78aaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510d001bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:50658/0 (socket says 172.21.15.94:50658) 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78aaa00640 1 -- 172.21.15.94:0/3666060667 learned_addr learned my addr 172.21.15.94:0/3666060667 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78ab400640 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78aaa00640 1 -- 172.21.15.94:0/3666060667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 msgr2=0x56510d0020f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78aaa00640 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 0x56510d0020f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78aaa00640 1 -- 172.21.15.94:0/3666060667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 msgr2=0x56510d005050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78aaa00640 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78aaa00640 1 -- 172.21.15.94:0/3666060667 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56510d005640 con 0x56510cff40f0 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.073+0000 7f78a2000640 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 0x56510d0020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.074+0000 7f78aaa00640 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510d001bb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f789c019040 tx=0x7f789c004310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=673ddb287f08c55a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.074+0000 7f788fe00640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.074+0000 7f788fe00640 1 -- 172.21.15.94:0/3666060667 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c039030 con 0x56510cff40f0 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.074+0000 7f78b43b2740 1 -- 172.21.15.94:0/3666060667 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f789c0039c0 con 0x56510cff40f0 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.074+0000 7f788fe00640 1 -- 172.21.15.94:0/3666060667 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f789c030070 con 0x56510cff40f0 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.074+0000 7f788fe00640 1 -- 172.21.15.94:0/3666060667 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f789c046050 con 0x56510cff40f0 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.074+0000 7f78b43b2740 10 client.25318 fetch_fsmap learned FSMap version 24 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.074+0000 7f78b43b2740 1 -- 172.21.15.94:0/3666060667 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56510d071500 con 0x56510cff40f0 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.074+0000 7f788fe00640 1 -- 172.21.15.94:0/3666060667 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f789c0039c0 con 0x56510cff40f0 2024-04-26T20:12:26.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.075+0000 7f788fe00640 1 -- 172.21.15.94:0/3666060667 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f789c0046c0 con 0x56510cff40f0 2024-04-26T20:12:26.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.075+0000 7f78b43b2740 10 client.25318 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:12:26.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.075+0000 7f78b43b2740 10 client.25318 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T20:12:26.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.075+0000 7f78b43b2740 20 client.25318 populate_metadata read hostname 'smithi094' 2024-04-26T20:12:26.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.075+0000 7f78b43b2740 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x56510d075e20 0x56510d072930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.075+0000 7f78b43b2740 4 client.25318 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:12:26.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.075+0000 7f78b43b2740 1 -- 172.21.15.94:0/3666060667 --> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] -- command(tid 0: {"prefix": "status"}) -- 0x56510cf60080 con 0x56510d075e20 2024-04-26T20:12:26.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.075+0000 7f78a2000640 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x56510d075e20 0x56510d072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f788fe00640 10 client.25318 ms_handle_connect on v2:172.21.15.204:6838/3633033106 2024-04-26T20:12:26.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78a2000640 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x56510d075e20 0x56510d072930 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=78a3540e733b7bae server_cookie=279e61137f75b613 in_seq=0 out_seq=0 2024-04-26T20:12:26.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f788fe00640 1 -- 172.21.15.94:0/3666060667 <== mds.2 v2:172.21.15.204:6838/3633033106 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x56510cf60080 con 0x56510d075e20 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f788fe00640 10 client.25318 handle_command_reply: tid=0 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78b43b2740 1 client.25318 shutdown 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78b43b2740 1 -- 172.21.15.94:0/3666060667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 msgr2=0x56510d001bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78b43b2740 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510d001bb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f789c019040 tx=0x7f789c004310 comp rx=0 tx=0).stop 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78b43b2740 1 -- 172.21.15.94:0/3666060667 shutdown_connections 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78b43b2740 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.204:6838/3633033106,v1:172.21.15.204:6839/3633033106] conn(0x56510d075e20 0x56510d072930 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78b43b2740 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56510cff9c50 0x56510d005050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78b43b2740 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56510cff4ae0 0x56510d0020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78b43b2740 1 --2- 172.21.15.94:0/3666060667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56510cff40f0 0x56510d001bb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f788fe00640 0 client.25318 ms_handle_reset on v2:172.21.15.204:6838/3633033106 2024-04-26T20:12:26.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.076+0000 7f78b43b2740 1 -- 172.21.15.94:0/3666060667 >> 172.21.15.94:0/3666060667 conn(0x56510cf88d10 msgr2=0x56510cff2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.077+0000 7f78b43b2740 1 -- 172.21.15.94:0/3666060667 shutdown_connections 2024-04-26T20:12:26.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.077+0000 7f78b43b2740 1 -- 172.21.15.94:0/3666060667 wait complete. 2024-04-26T20:12:26.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.077+0000 7f78b43b2740 20 client.25318 trim_cache size 0 max 16384 2024-04-26T20:12:26.082 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-26T20:12:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T20:12:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 2, 2024-04-26T20:12:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24541, 2024-04-26T20:12:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-26T20:12:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-26T20:12:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-26T20:12:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 182.73189278800001, 2024-04-26T20:12:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 24, 2024-04-26T20:12:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 96, 2024-04-26T20:12:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 94, 2024-04-26T20:12:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 275.59810303 2024-04-26T20:12:26.083 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.077+0000 7f78b1600640 1 -- 172.21.15.94:0/3890600695 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78900a3cc0 msgr2=0x7f78900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.077+0000 7f78b1600640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78900a3cc0 0x7f78900a6180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x56510cff4ee0 tx=0x7f789c005f90 comp rx=0 tx=0).stop 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.077+0000 7f78b1600640 1 -- 172.21.15.94:0/3890600695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10a1a0 msgr2=0x7f78ac19f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.077+0000 7f78b1600640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10a1a0 0x7f78ac19f640 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f789400e9f0 tx=0x7f789400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.078+0000 7f78b1600640 1 -- 172.21.15.94:0/3890600695 shutdown_connections 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.078+0000 7f78b1600640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f78900a3cc0 0x7f78900a6180 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.078+0000 7f78b1600640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78ac10ab60 0x7f78ac1a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.078+0000 7f78b1600640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78ac10a1a0 0x7f78ac19f640 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.078+0000 7f78b1600640 1 --2- 172.21.15.94:0/3890600695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78ac108fb0 0x7f78ac19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.078+0000 7f78b1600640 1 -- 172.21.15.94:0/3890600695 >> 172.21.15.94:0/3890600695 conn(0x7f78ac077f40 msgr2=0x7f78ac10d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.078+0000 7f78b1600640 1 -- 172.21.15.94:0/3890600695 shutdown_connections 2024-04-26T20:12:26.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.078+0000 7f78b1600640 1 -- 172.21.15.94:0/3890600695 wait complete. 2024-04-26T20:12:26.093 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:25 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1563350008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:12:26.093 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:25 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2288774682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:12:26.093 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:25 smithi094 ceph-mon[28050]: pgmap v453: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:26.097 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T20:12:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:25 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1563350008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:12:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:25 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2288774682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:12:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:25 smithi204 ceph-mon[32618]: pgmap v453: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:26.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:25 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1563350008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:12:26.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:25 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2288774682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:12:26.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:25 smithi103 ceph-mon[32635]: pgmap v453: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.294+0000 7f8fe0c00640 1 -- 172.21.15.94:0/2967806674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 msgr2=0x7f8fdc078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.294+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/2967806674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 0x7f8fdc078990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009860 tx=0x7f8fc802a340 comp rx=0 tx=0).stop 2024-04-26T20:12:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.294+0000 7f8fe0c00640 1 -- 172.21.15.94:0/2967806674 shutdown_connections 2024-04-26T20:12:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.294+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/2967806674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fdc078ed0 0x7f8fdc079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.294+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/2967806674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 0x7f8fdc078990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.294+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/2967806674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fdc0772f0 0x7f8fdc077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.294+0000 7f8fe0c00640 1 -- 172.21.15.94:0/2967806674 >> 172.21.15.94:0/2967806674 conn(0x7f8fdc100120 msgr2=0x7f8fdc102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.294+0000 7f8fe0c00640 1 -- 172.21.15.94:0/2967806674 shutdown_connections 2024-04-26T20:12:26.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.294+0000 7f8fe0c00640 1 -- 172.21.15.94:0/2967806674 wait complete. 2024-04-26T20:12:26.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.295+0000 7f8fe0c00640 1 Processor -- start 2024-04-26T20:12:26.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.295+0000 7f8fe0c00640 1 -- start start 2024-04-26T20:12:26.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.295+0000 7f8fe0c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fdc0772f0 0x7f8fdc19f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.295+0000 7f8fe0c00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 0x7f8fdc19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.295+0000 7f8fe0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fdc078ed0 0x7f8fdc1a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.295+0000 7f8fe0c00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc114af0 con 0x7f8fdc078530 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.295+0000 7f8fe0c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc114970 con 0x7f8fdc078ed0 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.295+0000 7f8fe0c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc114c70 con 0x7f8fdc0772f0 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 0x7f8fdc19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fdc0772f0 0x7f8fdc19f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 0x7f8fdc19f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46548/0 (socket says 172.21.15.94:46548) 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9400640 1 -- 172.21.15.94:0/1784779433 learned_addr learned my addr 172.21.15.94:0/1784779433 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fda800640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fdc078ed0 0x7f8fdc1a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9400640 1 -- 172.21.15.94:0/1784779433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fdc0772f0 msgr2=0x7f8fdc19f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9400640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fdc0772f0 0x7f8fdc19f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9400640 1 -- 172.21.15.94:0/1784779433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fdc078ed0 msgr2=0x7f8fdc1a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9400640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fdc078ed0 0x7f8fdc1a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9400640 1 -- 172.21.15.94:0/1784779433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fdc1a6dd0 con 0x7f8fdc078530 2024-04-26T20:12:26.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fda800640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fdc078ed0 0x7f8fdc1a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:12:26.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.296+0000 7f8fd9400640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 0x7f8fdc19f6f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80057c0 tx=0x7f8fc8032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.297+0000 7f8fd2a00640 1 -- 172.21.15.94:0/1784779433 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8037040 con 0x7f8fdc078530 2024-04-26T20:12:26.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.297+0000 7f8fd2a00640 1 -- 172.21.15.94:0/1784779433 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc8030bd0 con 0x7f8fdc078530 2024-04-26T20:12:26.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.297+0000 7f8fe0c00640 1 -- 172.21.15.94:0/1784779433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fdc1a7060 con 0x7f8fdc078530 2024-04-26T20:12:26.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.297+0000 7f8fd2a00640 1 -- 172.21.15.94:0/1784779433 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8005280 con 0x7f8fdc078530 2024-04-26T20:12:26.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.297+0000 7f8fe0c00640 1 -- 172.21.15.94:0/1784779433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fdc1a74f0 con 0x7f8fdc078530 2024-04-26T20:12:26.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.300+0000 7f8fe3a18740 1 Processor -- start 2024-04-26T20:12:26.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.300+0000 7f8fe3a18740 1 -- start start 2024-04-26T20:12:26.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.300+0000 7f8fd2a00640 1 -- 172.21.15.94:0/1784779433 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8fc8043020 con 0x7f8fdc078530 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1e4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1e95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e9c50 0x55878c1ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55878c148600 con 0x55878c1e40f0 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55878c0e5100 con 0x55878c1e4ae0 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55878c1445c0 con 0x55878c1e9c50 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fda800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fda800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1e4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46556/0 (socket says 172.21.15.94:46556) 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fda800640 1 -- 172.21.15.94:0/1598949291 learned_addr learned my addr 172.21.15.94:0/1598949291 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fd9400640 1 --2- 172.21.15.94:0/1598949291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e9c50 0x55878c1ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fd9e00640 1 --2- 172.21.15.94:0/1598949291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1e95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fda800640 1 -- 172.21.15.94:0/1598949291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e9c50 msgr2=0x55878c1ec0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fda800640 1 --2- 172.21.15.94:0/1598949291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e9c50 0x55878c1ec0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fda800640 1 -- 172.21.15.94:0/1598949291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 msgr2=0x55878c1e95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fda800640 1 --2- 172.21.15.94:0/1598949291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1e95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.301+0000 7f8fda800640 1 -- 172.21.15.94:0/1598949291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55878be5a2d0 con 0x55878c1e40f0 2024-04-26T20:12:26.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.302+0000 7f8fd2a00640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8fc00a3c40 0x7f8fc00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.302+0000 7f8fda800640 1 --2- 172.21.15.94:0/1598949291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1e4510 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400f480 tx=0x7f8fd400f940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0221b063db399c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.302+0000 7f8fd2a00640 1 -- 172.21.15.94:0/1784779433 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f8fc80effc0 con 0x7f8fdc078530 2024-04-26T20:12:26.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.302+0000 7f8fc7000640 1 -- 172.21.15.94:0/1598949291 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd401d070 con 0x55878c1e40f0 2024-04-26T20:12:26.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.302+0000 7f8fd9e00640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8fc00a3c40 0x7f8fc00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.302+0000 7f8fc7000640 1 -- 172.21.15.94:0/1598949291 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fd400cc30 con 0x55878c1e40f0 2024-04-26T20:12:26.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fc7000640 1 -- 172.21.15.94:0/1598949291 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4018cb0 con 0x55878c1e40f0 2024-04-26T20:12:26.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1598949291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 msgr2=0x55878c1e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1598949291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1e4510 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400f480 tx=0x7f8fd400f940 comp rx=0 tx=0).stop 2024-04-26T20:12:26.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fd9e00640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8fc00a3c40 0x7f8fc00a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55878c1e4ee0 tx=0x7f8fcc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1598949291 shutdown_connections 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1598949291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e9c50 0x55878c1ec0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1598949291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1e95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1598949291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1e4510 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1598949291 >> 172.21.15.94:0/1598949291 conn(0x55878c178d10 msgr2=0x55878c1dfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1598949291 shutdown_connections 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1598949291 wait complete. 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.303+0000 7f8fe3a18740 1 -- start start 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fe3a18740 1 Processor -- start 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1ee530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e4ae0 0x55878c1eea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e9c50 0x55878c1f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55878c1ecc50 con 0x55878c1e40f0 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55878c0cff70 con 0x55878c1e9c50 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55878c1ecdd0 con 0x55878c1e4ae0 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fe3a18740 10 client.0 _ll_register_callbacks cb 0x55878c13e480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:12:26.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fd9e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1ee530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fd9e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1ee530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46570/0 (socket says 172.21.15.94:46570) 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fd9e00640 1 -- 172.21.15.94:0/1858636604 learned_addr learned my addr 172.21.15.94:0/1858636604 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.304+0000 7f8fd9400640 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e4ae0 0x55878c1eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.305+0000 7f8fda800640 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e9c50 0x55878c1f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.305+0000 7f8fd9e00640 1 -- 172.21.15.94:0/1858636604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e4ae0 msgr2=0x55878c1eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.305+0000 7f8fd9e00640 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e4ae0 0x55878c1eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.305+0000 7f8fd9e00640 1 -- 172.21.15.94:0/1858636604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e9c50 msgr2=0x55878c1f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.305+0000 7f8fd9e00640 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e9c50 0x55878c1f5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.305+0000 7f8fd9e00640 1 -- 172.21.15.94:0/1858636604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55878bf246b0 con 0x55878c1e40f0 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.305+0000 7f8fd9e00640 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1ee530 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc01e9d0 tx=0x7f8fcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e72036524e48fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.305+0000 7f8fbb400640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.305+0000 7f8fbb400640 1 -- 172.21.15.94:0/1858636604 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc02e060 con 0x55878c1e40f0 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fbb400640 1 -- 172.21.15.94:0/1858636604 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f8fcc018030 con 0x55878c1e40f0 2024-04-26T20:12:26.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1858636604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fd400cc30 con 0x55878c1e40f0 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fbb400640 1 -- 172.21.15.94:0/1858636604 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8fcc035050 con 0x55878c1e40f0 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fe3a18740 10 client.15237 fetch_fsmap learned FSMap version 24 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1858636604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55878c1f8b60 con 0x55878c1e40f0 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fbb400640 1 -- 172.21.15.94:0/1858636604 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fd400cc30 con 0x55878c1e40f0 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fbb400640 1 -- 172.21.15.94:0/1858636604 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8fcc01c430 con 0x55878c1e40f0 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fe3a18740 10 client.15237 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fe3a18740 10 client.15237 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fe3a18740 20 client.15237 populate_metadata read hostname 'smithi094' 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.306+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x55878c1fb220 0x55878c1fc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.307+0000 7f8fe3a18740 4 client.15237 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.307+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1858636604 --> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55878c156210 con 0x55878c1fb220 2024-04-26T20:12:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.307+0000 7f8fd9400640 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x55878c1fb220 0x55878c1fc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.307+0000 7f8fbb400640 10 client.15237 ms_handle_connect on v2:172.21.15.94:6840/989331046 2024-04-26T20:12:26.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.307+0000 7f8fd9400640 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x55878c1fb220 0x55878c1fc690 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1bed020f5a90e547 server_cookie=b6be516eb6b21fba in_seq=0 out_seq=0 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.308+0000 7f8fbb400640 1 -- 172.21.15.94:0/1858636604 <== mds.1 v2:172.21.15.94:6840/989331046 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55878c156210 con 0x55878c1fb220 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.308+0000 7f8fbb400640 10 client.15237 handle_command_reply: tid=0 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.308+0000 7f8fe3a18740 1 client.15237 shutdown 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.308+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1858636604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 msgr2=0x55878c1ee530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.308+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1ee530 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc01e9d0 tx=0x7f8fcc004730 comp rx=0 tx=0).stop 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1858636604 shutdown_connections 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x55878c1fb220 0x55878c1fc690 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e9c50 0x55878c1f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e4ae0 0x55878c1eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fbb400640 0 client.15237 ms_handle_reset on v2:172.21.15.94:6840/989331046 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/1858636604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e40f0 0x55878c1ee530 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1858636604 >> 172.21.15.94:0/1858636604 conn(0x55878c178d10 msgr2=0x55878c1deeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1858636604 shutdown_connections 2024-04-26T20:12:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fe3a18740 1 -- 172.21.15.94:0/1858636604 wait complete. 2024-04-26T20:12:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.309+0000 7f8fe3a18740 20 client.15237 trim_cache size 0 max 16384 2024-04-26T20:12:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fe3a18740 1 Processor -- start 2024-04-26T20:12:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fe3a18740 1 -- start start 2024-04-26T20:12:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 0x55878c1e5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1e6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55878c1fa9d0 con 0x55878c1e9c50 2024-04-26T20:12:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55878c1fb0a0 con 0x55878c1e4ae0 2024-04-26T20:12:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55878c1fb770 con 0x55878c1e40f0 2024-04-26T20:12:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fda800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.329+0000 7f8fda800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1e6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46578/0 (socket says 172.21.15.94:46578) 2024-04-26T20:12:26.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fda800640 1 -- 172.21.15.94:0/3119969826 learned_addr learned my addr 172.21.15.94:0/3119969826 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fd9400640 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fd9e00640 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 0x55878c1e5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fda800640 1 -- 172.21.15.94:0/3119969826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 msgr2=0x55878c1e5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fda800640 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 0x55878c1e5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fda800640 1 -- 172.21.15.94:0/3119969826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 msgr2=0x55878c1f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fda800640 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fda800640 1 -- 172.21.15.94:0/3119969826 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55878c1db4b0 con 0x55878c1e9c50 2024-04-26T20:12:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fd9400640 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fda800640 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1e6380 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8fd401a000 tx=0x7f8fd40049e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c082371a4b91fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.330+0000 7f8fc6600640 1 -- 172.21.15.94:0/3119969826 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4027030 con 0x55878c1e9c50 2024-04-26T20:12:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fc6600640 1 -- 172.21.15.94:0/3119969826 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fd401e040 con 0x55878c1e9c50 2024-04-26T20:12:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fc6600640 1 -- 172.21.15.94:0/3119969826 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd401d050 con 0x55878c1e9c50 2024-04-26T20:12:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fe3a18740 1 -- 172.21.15.94:0/3119969826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 msgr2=0x55878c1e6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1e6380 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8fd401a000 tx=0x7f8fd40049e0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fe3a18740 1 -- 172.21.15.94:0/3119969826 shutdown_connections 2024-04-26T20:12:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1e6380 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/3119969826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 0x55878c1e5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fe3a18740 1 -- 172.21.15.94:0/3119969826 >> 172.21.15.94:0/3119969826 conn(0x55878c178d10 msgr2=0x55878c1fa020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fe3a18740 1 -- 172.21.15.94:0/3119969826 shutdown_connections 2024-04-26T20:12:26.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.331+0000 7f8fe3a18740 1 -- 172.21.15.94:0/3119969826 wait complete. 2024-04-26T20:12:26.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.332+0000 7f8fe3a18740 1 -- start start 2024-04-26T20:12:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.332+0000 7f8fe3a18740 1 Processor -- start 2024-04-26T20:12:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.332+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 0x55878c1f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.332+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.332+0000 7f8fe3a18740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.332+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x55878c1fc510 con 0x55878c1e9c50 2024-04-26T20:12:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.332+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55878c1fbe40 con 0x55878c1e4ae0 2024-04-26T20:12:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.332+0000 7f8fe3a18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55878c1fcbe0 con 0x55878c1e40f0 2024-04-26T20:12:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.332+0000 7f8fe3a18740 10 client.0 _ll_register_callbacks cb 0x7f8fcc009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:12:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fda800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fd9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fd9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 0x55878c1f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fda800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1f4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46580/0 (socket says 172.21.15.94:46580) 2024-04-26T20:12:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fda800640 1 -- 172.21.15.94:0/2795708489 learned_addr learned my addr 172.21.15.94:0/2795708489 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fd9400640 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:41796/0 (socket says 172.21.15.94:41796) 2024-04-26T20:12:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fd9400640 1 -- 172.21.15.94:0/2795708489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 msgr2=0x55878c1f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fd9400640 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 0x55878c1f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fd9400640 1 -- 172.21.15.94:0/2795708489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 msgr2=0x55878c1f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fd9400640 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1f4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fd9400640 1 -- 172.21.15.94:0/2795708489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55878c1f55e0 con 0x55878c1e4ae0 2024-04-26T20:12:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fda800640 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fd9400640 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f1fd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ed650 tx=0x7f8fc8005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26c6d4b374520176 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fbb400640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-26T20:12:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fbb400640 1 -- 172.21.15.94:0/2795708489 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80bb780 con 0x55878c1e4ae0 2024-04-26T20:12:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fe3a18740 1 -- 172.21.15.94:0/2795708489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fd401e040 con 0x55878c1e4ae0 2024-04-26T20:12:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.333+0000 7f8fbb400640 1 -- 172.21.15.94:0/2795708489 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f8fc8004000 con 0x55878c1e4ae0 2024-04-26T20:12:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fbb400640 1 -- 172.21.15.94:0/2795708489 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8fc80bc520 con 0x55878c1e4ae0 2024-04-26T20:12:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fe3a18740 10 client.25346 fetch_fsmap learned FSMap version 24 2024-04-26T20:12:26.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fe3a18740 1 -- 172.21.15.94:0/2795708489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55878c261a50 con 0x55878c1e4ae0 2024-04-26T20:12:26.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fbb400640 1 -- 172.21.15.94:0/2795708489 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fd401e040 con 0x55878c1e4ae0 2024-04-26T20:12:26.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fbb400640 1 -- 172.21.15.94:0/2795708489 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8fc80b8490 con 0x55878c1e4ae0 2024-04-26T20:12:26.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fe3a18740 10 client.25346 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:12:26.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fe3a18740 10 client.25346 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T20:12:26.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fe3a18740 20 client.25346 populate_metadata read hostname 'smithi094' 2024-04-26T20:12:26.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x55878c266670 0x55878c262ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fe3a18740 4 client.25346 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:12:26.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fe3a18740 1 -- 172.21.15.94:0/2795708489 --> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] -- command(tid 0: {"prefix": "status"}) -- 0x55878c150080 con 0x55878c266670 2024-04-26T20:12:26.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.334+0000 7f8fd9e00640 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x55878c266670 0x55878c262ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fd9e00640 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x55878c266670 0x55878c262ab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dc20f2727d9a1ee2 server_cookie=2956466f18fbd6c9 in_seq=0 out_seq=0 2024-04-26T20:12:26.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fbb400640 10 client.25346 ms_handle_connect on v2:172.21.15.94:6840/989331046 2024-04-26T20:12:26.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fbb400640 1 -- 172.21.15.94:0/2795708489 <== mds.1 v2:172.21.15.94:6840/989331046 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55878c150080 con 0x55878c266670 2024-04-26T20:12:26.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fbb400640 10 client.25346 handle_command_reply: tid=0 2024-04-26T20:12:26.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fe3a18740 1 client.25346 shutdown 2024-04-26T20:12:26.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fe3a18740 1 -- 172.21.15.94:0/2795708489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 msgr2=0x55878c1f1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f1fd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ed650 tx=0x7f8fc8005bd0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fe3a18740 1 -- 172.21.15.94:0/2795708489 shutdown_connections 2024-04-26T20:12:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.94:6840/989331046,v1:172.21.15.94:6841/989331046] conn(0x55878c266670 0x55878c262ab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55878c1e9c50 0x55878c1f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55878c1e4ae0 0x55878c1f1fd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fe3a18740 1 --2- 172.21.15.94:0/2795708489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55878c1e40f0 0x55878c1f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fbb400640 0 client.25346 ms_handle_reset on v2:172.21.15.94:6840/989331046 2024-04-26T20:12:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.335+0000 7f8fe3a18740 1 -- 172.21.15.94:0/2795708489 >> 172.21.15.94:0/2795708489 conn(0x55878c178d10 msgr2=0x55878c1e5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe3a18740 1 -- 172.21.15.94:0/2795708489 shutdown_connections 2024-04-26T20:12:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe3a18740 1 -- 172.21.15.94:0/2795708489 wait complete. 2024-04-26T20:12:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe3a18740 20 client.25346 trim_cache size 0 max 16384 2024-04-26T20:12:26.355 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-26T20:12:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T20:12:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 1, 2024-04-26T20:12:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24562, 2024-04-26T20:12:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-26T20:12:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-26T20:12:26.356 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-26T20:12:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 184.00425812700001, 2024-04-26T20:12:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 24, 2024-04-26T20:12:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 96, 2024-04-26T20:12:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 94, 2024-04-26T20:12:26.357 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 267.42720401700001 2024-04-26T20:12:26.357 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-26T20:12:26.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 -- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8fc00a3c40 msgr2=0x7f8fc00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8fc00a3c40 0x7f8fc00a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55878c1e4ee0 tx=0x7f8fcc005f90 comp rx=0 tx=0).stop 2024-04-26T20:12:26.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 -- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 msgr2=0x7f8fdc19f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 0x7f8fdc19f6f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80057c0 tx=0x7f8fc8032da0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 -- 172.21.15.94:0/1784779433 shutdown_connections 2024-04-26T20:12:26.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f8fc00a3c40 0x7f8fc00a6100 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fdc078ed0 0x7f8fdc1a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fdc078530 0x7f8fdc19f6f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 --2- 172.21.15.94:0/1784779433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8fdc0772f0 0x7f8fdc19f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.336+0000 7f8fe0c00640 1 -- 172.21.15.94:0/1784779433 >> 172.21.15.94:0/1784779433 conn(0x7f8fdc100120 msgr2=0x7f8fdc1018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.337+0000 7f8fe0c00640 1 -- 172.21.15.94:0/1784779433 shutdown_connections 2024-04-26T20:12:26.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.337+0000 7f8fe0c00640 1 -- 172.21.15.94:0/1784779433 wait complete. 2024-04-26T20:12:26.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T20:12:26.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.548+0000 7f2713e00640 1 -- 172.21.15.94:0/3006561154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714105b10 msgr2=0x7f2714107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.548+0000 7f2713e00640 1 --2- 172.21.15.94:0/3006561154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714105b10 0x7f2714107f00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009900 tx=0x7f26fc02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.548+0000 7f2713e00640 1 -- 172.21.15.94:0/3006561154 shutdown_connections 2024-04-26T20:12:26.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.548+0000 7f2713e00640 1 --2- 172.21.15.94:0/3006561154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2714108440 0x7f271410a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.548+0000 7f2713e00640 1 --2- 172.21.15.94:0/3006561154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714105b10 0x7f2714107f00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.548+0000 7f2713e00640 1 --2- 172.21.15.94:0/3006561154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27141031e0 0x7f27141055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.548+0000 7f2713e00640 1 -- 172.21.15.94:0/3006561154 >> 172.21.15.94:0/3006561154 conn(0x7f27140fcbb0 msgr2=0x7f27140ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.548+0000 7f2713e00640 1 -- 172.21.15.94:0/3006561154 shutdown_connections 2024-04-26T20:12:26.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.548+0000 7f2713e00640 1 -- 172.21.15.94:0/3006561154 wait complete. 2024-04-26T20:12:26.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.549+0000 7f2713e00640 1 Processor -- start 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.549+0000 7f2713e00640 1 -- start start 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.549+0000 7f2713e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27141031e0 0x7f271419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.549+0000 7f2713e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2714105b10 0x7f271419b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.549+0000 7f2713e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714108440 0x7f27141a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.549+0000 7f2713e00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f271410e2f0 con 0x7f2714108440 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f271410e170 con 0x7f27141031e0 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f271410e470 con 0x7f2714105b10 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714108440 0x7f27141a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2712a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27141031e0 0x7f271419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714108440 0x7f27141a21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46586/0 (socket says 172.21.15.94:46586) 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2712a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27141031e0 0x7f271419abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:41806/0 (socket says 172.21.15.94:41806) 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713400640 1 -- 172.21.15.94:0/2529743592 learned_addr learned my addr 172.21.15.94:0/2529743592 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2712000640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2714105b10 0x7f271419b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713400640 1 -- 172.21.15.94:0/2529743592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2714105b10 msgr2=0x7f271419b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713400640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2714105b10 0x7f271419b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713400640 1 -- 172.21.15.94:0/2529743592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27141031e0 msgr2=0x7f271419abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713400640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27141031e0 0x7f271419abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713400640 1 -- 172.21.15.94:0/2529743592 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27141a2810 con 0x7f2714108440 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.550+0000 7f2713400640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714108440 0x7f27141a21a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f270c00e9b0 tx=0x7f270c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.551+0000 7f2718a00640 1 -- 172.21.15.94:0/2529743592 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c00cd80 con 0x7f2714108440 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.551+0000 7f2718a00640 1 -- 172.21.15.94:0/2529743592 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f270c0035f0 con 0x7f2714108440 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.551+0000 7f2718a00640 1 -- 172.21.15.94:0/2529743592 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c010690 con 0x7f2714108440 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.551+0000 7f2713e00640 1 -- 172.21.15.94:0/2529743592 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27141a2b00 con 0x7f2714108440 2024-04-26T20:12:26.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.551+0000 7f2713e00640 1 -- 172.21.15.94:0/2529743592 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27141a2fd0 con 0x7f2714108440 2024-04-26T20:12:26.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.554+0000 7f271c922740 1 Processor -- start 2024-04-26T20:12:26.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.554+0000 7f2718a00640 1 -- 172.21.15.94:0/2529743592 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f270c004020 con 0x7f2714108440 2024-04-26T20:12:26.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.554+0000 7f271c922740 1 -- start start 2024-04-26T20:12:26.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.554+0000 7f271c922740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcdd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f271c922740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fcddcb0 0x56273fce27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f271c922740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fce2e20 0x56273fce52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f271c922740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56273fc41600 con 0x56273fce2e20 2024-04-26T20:12:26.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f271c922740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56273fbde100 con 0x56273fcdd2c0 2024-04-26T20:12:26.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f271c922740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56273fc3d5c0 con 0x56273fcddcb0 2024-04-26T20:12:26.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcdd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fce2e20 0x56273fce52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fcddcb0 0x56273fce27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fcddcb0 0x56273fce27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:50744/0 (socket says 172.21.15.94:50744) 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712a00640 1 -- 172.21.15.94:0/4050499110 learned_addr learned my addr 172.21.15.94:0/4050499110 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712a00640 1 -- 172.21.15.94:0/4050499110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 msgr2=0x56273fcdd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712a00640 1 --2- 172.21.15.94:0/4050499110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcdd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712a00640 1 -- 172.21.15.94:0/4050499110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fce2e20 msgr2=0x56273fce52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712a00640 1 --2- 172.21.15.94:0/4050499110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fce2e20 0x56273fce52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.555+0000 7f2712a00640 1 -- 172.21.15.94:0/4050499110 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273f9532d0 con 0x56273fcddcb0 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.556+0000 7f2718a00640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26f80a3cc0 0x7f26f80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.556+0000 7f2712a00640 1 --2- 172.21.15.94:0/4050499110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fcddcb0 0x56273fce27b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f27040114f0 tx=0x7f27040119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1be8b681a6a9a5dc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.556+0000 7f2718a00640 1 -- 172.21.15.94:0/2529743592 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f270c0d0be0 con 0x7f2714108440 2024-04-26T20:12:26.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.556+0000 7f2703e00640 1 -- 172.21.15.94:0/4050499110 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704004d20 con 0x56273fcddcb0 2024-04-26T20:12:26.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.556+0000 7f2703e00640 1 -- 172.21.15.94:0/4050499110 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2704004050 con 0x56273fcddcb0 2024-04-26T20:12:26.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.556+0000 7f2703e00640 1 -- 172.21.15.94:0/4050499110 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270400f7c0 con 0x56273fcddcb0 2024-04-26T20:12:26.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.556+0000 7f2712000640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26f80a3cc0 0x7f26f80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 -- 172.21.15.94:0/4050499110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fcddcb0 msgr2=0x56273fce27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f2712000640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26f80a3cc0 0x7f26f80a6180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f26fc005ab0 tx=0x7f26fc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 --2- 172.21.15.94:0/4050499110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fcddcb0 0x56273fce27b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f27040114f0 tx=0x7f27040119b0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 -- 172.21.15.94:0/4050499110 shutdown_connections 2024-04-26T20:12:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 --2- 172.21.15.94:0/4050499110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fce2e20 0x56273fce52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 --2- 172.21.15.94:0/4050499110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fcddcb0 0x56273fce27b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 --2- 172.21.15.94:0/4050499110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcdd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 -- 172.21.15.94:0/4050499110 >> 172.21.15.94:0/4050499110 conn(0x56273fc71d10 msgr2=0x56273fcd8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 -- 172.21.15.94:0/4050499110 shutdown_connections 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 -- 172.21.15.94:0/4050499110 wait complete. 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 -- start start 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.557+0000 7f271c922740 1 Processor -- start 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f271c922740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fce7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f271c922740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fce7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f271c922740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fceecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f271c922740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56273fce5dd0 con 0x56273fcddcb0 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f271c922740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56273fbc8f70 con 0x56273fcdd2c0 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f271c922740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56273fce5f50 con 0x56273fce2e20 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f271c922740 10 client.0 _ll_register_callbacks cb 0x56273fc37480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2712000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fce7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2712000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fce7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46592/0 (socket says 172.21.15.94:46592) 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2712a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fce7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2712000640 1 -- 172.21.15.94:0/2088557084 learned_addr learned my addr 172.21.15.94:0/2088557084 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2713400640 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fceecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2712000640 1 -- 172.21.15.94:0/2088557084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 msgr2=0x56273fceecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2712000640 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fceecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2712000640 1 -- 172.21.15.94:0/2088557084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 msgr2=0x56273fce7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2712000640 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fce7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.558+0000 7f2712000640 1 -- 172.21.15.94:0/2088557084 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273fa1d6b0 con 0x56273fcddcb0 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.559+0000 7f2712000640 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fce7c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009900 tx=0x7f26fc02a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c82664d780b99cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.559+0000 7f2700c00640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.559+0000 7f2700c00640 1 -- 172.21.15.94:0/2088557084 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc035040 con 0x56273fcddcb0 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.559+0000 7f2700c00640 1 -- 172.21.15.94:0/2088557084 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f26fc0319b0 con 0x56273fcddcb0 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.559+0000 7f271c922740 1 -- 172.21.15.94:0/2088557084 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2704004050 con 0x56273fcddcb0 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.559+0000 7f2700c00640 1 -- 172.21.15.94:0/2088557084 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f26fc031bb0 con 0x56273fcddcb0 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.559+0000 7f271c922740 10 client.15255 fetch_fsmap learned FSMap version 24 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.559+0000 7f271c922740 1 -- 172.21.15.94:0/2088557084 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56273fcf1d40 con 0x56273fcddcb0 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.560+0000 7f2700c00640 1 -- 172.21.15.94:0/2088557084 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2704004050 con 0x56273fcddcb0 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.560+0000 7f2700c00640 1 -- 172.21.15.94:0/2088557084 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f26fc04d420 con 0x56273fcddcb0 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.560+0000 7f271c922740 10 client.15255 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.560+0000 7f271c922740 10 client.15255 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.560+0000 7f271c922740 20 client.15255 populate_metadata read hostname 'smithi094' 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.560+0000 7f271c922740 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x56273fcf4460 0x56273fcf58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.560+0000 7f271c922740 4 client.15255 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:12:26.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.560+0000 7f271c922740 1 -- 172.21.15.94:0/2088557084 --> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56273fc4f210 con 0x56273fcf4460 2024-04-26T20:12:26.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.561+0000 7f2712a00640 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x56273fcf4460 0x56273fcf58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.561+0000 7f2712a00640 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x56273fcf4460 0x56273fcf58d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfe771eae835845 server_cookie=72f8efde9eb9872 in_seq=0 out_seq=0 2024-04-26T20:12:26.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.561+0000 7f2700c00640 10 client.15255 ms_handle_connect on v2:172.21.15.103:6838/1048742250 2024-04-26T20:12:26.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.562+0000 7f2700c00640 1 -- 172.21.15.94:0/2088557084 <== mds.0 v2:172.21.15.103:6838/1048742250 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56273fc4f210 con 0x56273fcf4460 2024-04-26T20:12:26.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.562+0000 7f2700c00640 10 client.15255 handle_command_reply: tid=0 2024-04-26T20:12:26.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.562+0000 7f271c922740 1 client.15255 shutdown 2024-04-26T20:12:26.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 -- 172.21.15.94:0/2088557084 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 msgr2=0x56273fce7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fce7c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009900 tx=0x7f26fc02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 -- 172.21.15.94:0/2088557084 shutdown_connections 2024-04-26T20:12:26.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x56273fcf4460 0x56273fcf58d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fceecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fce7c80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f2700c00640 0 client.15255 ms_handle_reset on v2:172.21.15.103:6838/1048742250 2024-04-26T20:12:26.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 --2- 172.21.15.94:0/2088557084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fce7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 -- 172.21.15.94:0/2088557084 >> 172.21.15.94:0/2088557084 conn(0x56273fc71d10 msgr2=0x56273fcd7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 -- 172.21.15.94:0/2088557084 shutdown_connections 2024-04-26T20:12:26.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 1 -- 172.21.15.94:0/2088557084 wait complete. 2024-04-26T20:12:26.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.563+0000 7f271c922740 20 client.15255 trim_cache size 0 max 16384 2024-04-26T20:12:26.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.583+0000 7f271c922740 1 Processor -- start 2024-04-26T20:12:26.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.583+0000 7f271c922740 1 -- start start 2024-04-26T20:12:26.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f271c922740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fce7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f271c922740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f271c922740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fce6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f271c922740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56273fcf3c10 con 0x56273fcddcb0 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f271c922740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56273fcf42e0 con 0x56273fcdd2c0 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f271c922740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56273fcf49b0 con 0x56273fce2e20 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2712000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2712000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46602/0 (socket says 172.21.15.94:46602) 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2712000640 1 -- 172.21.15.94:0/1814424288 learned_addr learned my addr 172.21.15.94:0/1814424288 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2712a00640 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fce7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2713400640 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fce6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2712000640 1 -- 172.21.15.94:0/1814424288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 msgr2=0x56273fce6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2712000640 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fce6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2712000640 1 -- 172.21.15.94:0/1814424288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 msgr2=0x56273fce7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2712000640 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fce7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2712000640 1 -- 172.21.15.94:0/1814424288 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273fce2380 con 0x56273fcddcb0 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.584+0000 7f2713400640 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fce6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.585+0000 7f2712000640 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceec80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f26fc045040 tx=0x7f26fc003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451066265b04466c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.585+0000 7f2703400640 1 -- 172.21.15.94:0/1814424288 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc035070 con 0x56273fcddcb0 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.585+0000 7f2703400640 1 -- 172.21.15.94:0/1814424288 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26fc0317b0 con 0x56273fcddcb0 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.585+0000 7f2703400640 1 -- 172.21.15.94:0/1814424288 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc0558b0 con 0x56273fcddcb0 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 -- 172.21.15.94:0/1814424288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 msgr2=0x56273fceec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceec80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f26fc045040 tx=0x7f26fc003f40 comp rx=0 tx=0).stop 2024-04-26T20:12:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 -- 172.21.15.94:0/1814424288 shutdown_connections 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fce6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceec80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 --2- 172.21.15.94:0/1814424288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fce7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 -- 172.21.15.94:0/1814424288 >> 172.21.15.94:0/1814424288 conn(0x56273fc71d10 msgr2=0x56273fce7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 -- 172.21.15.94:0/1814424288 shutdown_connections 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 -- 172.21.15.94:0/1814424288 wait complete. 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 -- start start 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.586+0000 7f271c922740 1 Processor -- start 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f271c922740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcea920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f271c922740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f271c922740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fcede10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f271c922740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x56273fcf5750 con 0x56273fcddcb0 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f271c922740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56273fcf5080 con 0x56273fcdd2c0 2024-04-26T20:12:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f271c922740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56273fcf5e20 con 0x56273fce2e20 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f271c922740 10 client.0 _ll_register_callbacks cb 0x7f26fc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcea920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46612/0 (socket says 172.21.15.94:46612) 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712000640 1 -- 172.21.15.94:0/3048983416 learned_addr learned my addr 172.21.15.94:0/3048983416 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcea920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:41842/0 (socket says 172.21.15.94:41842) 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2713400640 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fcede10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712000640 1 -- 172.21.15.94:0/3048983416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 msgr2=0x56273fcede10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712000640 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fcede10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712000640 1 -- 172.21.15.94:0/3048983416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 msgr2=0x56273fcea920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712000640 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcea920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712000640 1 -- 172.21.15.94:0/3048983416 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273fcee400 con 0x56273fcddcb0 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.587+0000 7f2712a00640 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcea920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.588+0000 7f2712000640 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceae60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f26fc003fd0 tx=0x7f26fc0092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfdd2863a19baae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.588+0000 7f2700c00640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.588+0000 7f2700c00640 1 -- 172.21.15.94:0/3048983416 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc035040 con 0x56273fcddcb0 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.588+0000 7f2700c00640 1 -- 172.21.15.94:0/3048983416 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f26fc062680 con 0x56273fcddcb0 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.588+0000 7f271c922740 1 -- 172.21.15.94:0/3048983416 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26fc0317b0 con 0x56273fcddcb0 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.588+0000 7f2700c00640 1 -- 172.21.15.94:0/3048983416 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f26fc062880 con 0x56273fcddcb0 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.588+0000 7f271c922740 10 client.15261 fetch_fsmap learned FSMap version 24 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.588+0000 7f271c922740 1 -- 172.21.15.94:0/3048983416 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56273fd5a8f0 con 0x56273fcddcb0 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.589+0000 7f2700c00640 1 -- 172.21.15.94:0/3048983416 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26fc0317b0 con 0x56273fcddcb0 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.589+0000 7f2700c00640 1 -- 172.21.15.94:0/3048983416 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f26fc03db30 con 0x56273fcddcb0 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.589+0000 7f271c922740 10 client.15261 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.589+0000 7f271c922740 10 client.15261 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.589+0000 7f271c922740 20 client.15261 populate_metadata read hostname 'smithi094' 2024-04-26T20:12:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.589+0000 7f271c922740 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x56273fd5f060 0x56273fd5bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.589+0000 7f271c922740 4 client.15261 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.589+0000 7f271c922740 1 -- 172.21.15.94:0/3048983416 --> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] -- command(tid 0: {"prefix": "status"}) -- 0x56273fc49080 con 0x56273fd5f060 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.589+0000 7f2712a00640 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x56273fd5f060 0x56273fd5bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f2712a00640 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x56273fd5f060 0x56273fd5bb70 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6614aa0794fcad9f server_cookie=e1e9192f15480e73 in_seq=0 out_seq=0 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f2700c00640 10 client.15261 ms_handle_connect on v2:172.21.15.103:6838/1048742250 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f2700c00640 1 -- 172.21.15.94:0/3048983416 <== mds.0 v2:172.21.15.103:6838/1048742250 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56273fc49080 con 0x56273fd5f060 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f2700c00640 10 client.15261 handle_command_reply: tid=0 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 client.15261 shutdown 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 -- 172.21.15.94:0/3048983416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 msgr2=0x56273fceae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceae60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f26fc003fd0 tx=0x7f26fc0092d0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 -- 172.21.15.94:0/3048983416 shutdown_connections 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.103:6838/1048742250,v1:172.21.15.103:6839/1048742250] conn(0x56273fd5f060 0x56273fd5bb70 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56273fce2e20 0x56273fcede10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56273fcddcb0 0x56273fceae60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 --2- 172.21.15.94:0/3048983416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56273fcdd2c0 0x56273fcea920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 -- 172.21.15.94:0/3048983416 >> 172.21.15.94:0/3048983416 conn(0x56273fc71d10 msgr2=0x56273fce6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f2700c00640 0 client.15261 ms_handle_reset on v2:172.21.15.103:6838/1048742250 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.590+0000 7f271c922740 1 -- 172.21.15.94:0/3048983416 shutdown_connections 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f271c922740 1 -- 172.21.15.94:0/3048983416 wait complete. 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f271c922740 20 client.15261 trim_cache size 0 max 16384 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "420d0bd6-0407-11ef-bc93-c7b262605968", 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 24575, 2024-04-26T20:12:26.600 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-04-26T20:12:26.601 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-04-26T20:12:26.601 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-04-26T20:12:26.601 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 247.93295899699999, 2024-04-26T20:12:26.601 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 24, 2024-04-26T20:12:26.601 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 96, 2024-04-26T20:12:26.601 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 94, 2024-04-26T20:12:26.601 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 264.85946072299998 2024-04-26T20:12:26.601 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 -- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26f80a3cc0 msgr2=0x7f26f80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26f80a3cc0 0x7f26f80a6180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f26fc005ab0 tx=0x7f26fc005a20 comp rx=0 tx=0).stop 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 -- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714108440 msgr2=0x7f27141a21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714108440 0x7f27141a21a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f270c00e9b0 tx=0x7f270c00ee70 comp rx=0 tx=0).stop 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 -- 172.21.15.94:0/2529743592 shutdown_connections 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f26f80a3cc0 0x7f26f80a6180 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2714108440 0x7f27141a21a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2714105b10 0x7f271419b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 --2- 172.21.15.94:0/2529743592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27141031e0 0x7f271419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 -- 172.21.15.94:0/2529743592 >> 172.21.15.94:0/2529743592 conn(0x7f27140fcbb0 msgr2=0x7f2714102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:26.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.591+0000 7f2713e00640 1 -- 172.21.15.94:0/2529743592 shutdown_connections 2024-04-26T20:12:26.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:26.592+0000 7f2713e00640 1 -- 172.21.15.94:0/2529743592 wait complete. 2024-04-26T20:12:26.608 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T20:12:26.621 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T20:12:26.622 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T20:12:26.622 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T20:12:26.622 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T20:12:26.645 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T20:12:26.645 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T20:12:26.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:26.646 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-04-26T20:12:26.695 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-26T20:12:26.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:26.696 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:12:26.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:26.828 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2024-04-26T20:12:26.890 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:26 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3943343417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:12:26.935 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-26T20:12:26.935 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T20:12:26.935 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T20:12:26.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T20:12:27.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:26 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3943343417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:12:27.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:26 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3943343417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:12:27.355 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:12:28.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:27 smithi204 ceph-mon[32618]: pgmap v454: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:27 smithi094 ceph-mon[28050]: pgmap v454: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:27 smithi103 ceph-mon[32635]: pgmap v454: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:29.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.555+0000 7fc8b3200700 1 -- 172.21.15.94:0/2012866466 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8ac1094a0 msgr2=0x7fc8ac109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:29.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.555+0000 7fc8b3200700 1 --2- 172.21.15.94:0/2012866466 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8ac1094a0 0x7fc8ac109880 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc89c009990 tx=0x7fc89c009ca0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.555+0000 7fc8b3200700 1 -- 172.21.15.94:0/2012866466 shutdown_connections 2024-04-26T20:12:29.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.555+0000 7fc8b3200700 1 --2- 172.21.15.94:0/2012866466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8ac06bad0 0x7fc8ac1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.555+0000 7fc8b3200700 1 --2- 172.21.15.94:0/2012866466 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8ac1094a0 0x7fc8ac109880 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.555+0000 7fc8b3200700 1 --2- 172.21.15.94:0/2012866466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ac06b130 0x7fc8ac06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.555+0000 7fc8b3200700 1 -- 172.21.15.94:0/2012866466 >> 172.21.15.94:0/2012866466 conn(0x7fc8ac0792d0 msgr2=0x7fc8ac0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:29.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.555+0000 7fc8b3200700 1 -- 172.21.15.94:0/2012866466 shutdown_connections 2024-04-26T20:12:29.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.555+0000 7fc8b3200700 1 -- 172.21.15.94:0/2012866466 wait complete. 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.556+0000 7fc8b3200700 1 Processor -- start 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.556+0000 7fc8b3200700 1 -- start start 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8b3200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8ac06b130 0x7fc8ac1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8b3200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8ac06bad0 0x7fc8ac102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8b3200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ac1094a0 0x7fc8ac102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8b3200700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ac1135e0 con 0x7fc8ac1094a0 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8b3200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ac113460 con 0x7fc8ac06bad0 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8b3200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ac113760 con 0x7fc8ac06b130 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8b1200700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ac1094a0 0x7fc8ac102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8ab400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8ac06bad0 0x7fc8ac102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8ab400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8ac06bad0 0x7fc8ac102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:41852/0 (socket says 172.21.15.94:41852) 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8abe00700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8ac06b130 0x7fc8ac1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.557+0000 7fc8ab400700 1 -- 172.21.15.94:0/4008252907 learned_addr learned my addr 172.21.15.94:0/4008252907 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:29.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8ab400700 1 -- 172.21.15.94:0/4008252907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8ac06b130 msgr2=0x7fc8ac1023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:29.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8ab400700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8ac06b130 0x7fc8ac1023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8ab400700 1 -- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ac1094a0 msgr2=0x7fc8ac102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:29.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8ab400700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ac1094a0 0x7fc8ac102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8ab400700 1 -- 172.21.15.94:0/4008252907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ac1a80c0 con 0x7fc8ac06bad0 2024-04-26T20:12:29.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8b1200700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ac1094a0 0x7fc8ac102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:29.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8ab400700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8ac06bad0 0x7fc8ac102930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc89800eb30 tx=0x7fc89800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:29.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8a8c00700 1 -- 172.21.15.94:0/4008252907 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89800cbd0 con 0x7fc8ac06bad0 2024-04-26T20:12:29.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8ac1a83b0 con 0x7fc8ac06bad0 2024-04-26T20:12:29.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8a8c00700 1 -- 172.21.15.94:0/4008252907 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8980044d0 con 0x7fc8ac06bad0 2024-04-26T20:12:29.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8a8c00700 1 -- 172.21.15.94:0/4008252907 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc898003df0 con 0x7fc8ac06bad0 2024-04-26T20:12:29.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.558+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8ac1a88f0 con 0x7fc8ac06bad0 2024-04-26T20:12:29.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.559+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8ac06a110 con 0x7fc8ac06bad0 2024-04-26T20:12:29.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.562+0000 7fc8a8c00700 1 -- 172.21.15.94:0/4008252907 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc898004000 con 0x7fc8ac06bad0 2024-04-26T20:12:29.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.563+0000 7fc8a8c00700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc8900a3cd0 0x7fc8900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:29.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.563+0000 7fc8a8c00700 1 -- 172.21.15.94:0/4008252907 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fc8980d0c10 con 0x7fc8ac06bad0 2024-04-26T20:12:29.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.564+0000 7fc8abe00700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc8900a3cd0 0x7fc8900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:29.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.564+0000 7fc8abe00700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc8900a3cd0 0x7fc8900a6190 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00b2e0 tx=0x7fc89c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:29.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.568+0000 7fc8a8c00700 1 -- 172.21.15.94:0/4008252907 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc8980042f0 con 0x7fc8ac06bad0 2024-04-26T20:12:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.840+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc8ac02e490 con 0x7fc8ac06bad0 2024-04-26T20:12:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.840+0000 7fc8a8c00700 1 -- 172.21.15.94:0/4008252907 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7fc89809d060 con 0x7fc8ac06bad0 2024-04-26T20:12:29.842 INFO:teuthology.orchestra.run.smithi094.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T20:12:29.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc8900a3cd0 msgr2=0x7fc8900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:29.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc8900a3cd0 0x7fc8900a6190 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00b2e0 tx=0x7fc89c005fd0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8ac06bad0 msgr2=0x7fc8ac102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8ac06bad0 0x7fc8ac102930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc89800eb30 tx=0x7fc89800ee40 comp rx=0 tx=0).stop 2024-04-26T20:12:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 shutdown_connections 2024-04-26T20:12:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ac1094a0 0x7fc8ac102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8ac06b130 0x7fc8ac1023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc8900a3cd0 0x7fc8900a6190 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 --2- 172.21.15.94:0/4008252907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8ac06bad0 0x7fc8ac102930 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 >> 172.21.15.94:0/4008252907 conn(0x7fc8ac0792d0 msgr2=0x7fc8ac100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 shutdown_connections 2024-04-26T20:12:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:29.844+0000 7fc8b3200700 1 -- 172.21.15.94:0/4008252907 wait complete. 2024-04-26T20:12:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:30 smithi094 ceph-mon[28050]: pgmap v455: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:12:30.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:30 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4008252907' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:12:31.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:30 smithi103 ceph-mon[32635]: pgmap v455: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:12:31.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:30 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4008252907' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:12:31.062 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T20:12:31.062 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T20:12:31.062 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi094.front.sepia.ceph.com 2024-04-26T20:12:31.062 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T20:12:31.062 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T20:12:31.062 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T20:12:31.062 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T20:12:31.062 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T20:12:31.062 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T20:12:31.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:31.063 DEBUG:teuthology.orchestra.run.smithi094:> ip addr 2024-04-26T20:12:31.119 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T20:12:31.119 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T20:12:31.119 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T20:12:31.119 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:12:31.119 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2024-04-26T20:12:31.119 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:12:31.119 INFO:teuthology.orchestra.run.smithi094.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:12:31.119 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4c brd ff:ff:ff:ff:ff:ff 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f0 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4d brd ff:ff:ff:ff:ff:ff 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f1 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:62 brd ff:ff:ff:ff:ff:ff 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:63 brd ff:ff:ff:ff:ff:ff 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout: inet 172.21.15.94/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft 41822sec preferred_lft 41822sec 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::ec4:7aff:fe8f:d763/64 scope link 2024-04-26T20:12:31.120 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:12:31.121 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T20:12:31.121 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:12:31.122 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:12:31.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add name ceph-brx type bridge 2024-04-26T20:12:31.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr flush dev ceph-brx 2024-04-26T20:12:31.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set ceph-brx up 2024-04-26T20:12:31.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T20:12:31.122 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:12:31.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:30 smithi204 ceph-mon[32618]: pgmap v455: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:12:31.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:30 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4008252907' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:12:31.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:12:31.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:12:31.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:31.361 DEBUG:teuthology.orchestra.run.smithi094:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T20:12:31.426 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-04-26T20:12:31.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:31.430 DEBUG:teuthology.orchestra.run.smithi094:> route 2024-04-26T20:12:31.478 INFO:teuthology.orchestra.run.smithi094.stdout:Kernel IP routing table 2024-04-26T20:12:31.478 INFO:teuthology.orchestra.run.smithi094.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T20:12:31.478 INFO:teuthology.orchestra.run.smithi094.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-26T20:12:31.479 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-26T20:12:31.479 INFO:teuthology.orchestra.run.smithi094.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T20:12:31.479 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:12:31.479 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:12:31.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-26T20:12:31.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-26T20:12:31.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-26T20:12:31.479 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:12:31.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:12:31.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:12:31.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:31.687 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-04-26T20:12:31.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:31.737 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list-id 2024-04-26T20:12:31.789 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:12:31.789 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:12:31.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T20:12:31.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-26T20:12:31.789 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:12:31.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:12:31.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:12:31.973 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-26T20:12:31.974 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:12:31.974 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:12:31.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-26T20:12:31.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T20:12:31.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-26T20:12:31.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-26T20:12:31.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-26T20:12:31.974 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:12:32.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:12:32.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:12:32.244 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:12:32.244 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-04-26T20:12:32.244 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set brx.0 up 2024-04-26T20:12:32.244 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T20:12:32.244 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-04-26T20:12:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:12:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:12:32.444 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T20:12:32.444 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T20:12:32.491 INFO:teuthology.orchestra.run.smithi094.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T20:12:32.491 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T20:12:32.491 DEBUG:teuthology.orchestra.run.smithi094:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T20:12:32.588 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T20:12:32.588 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T20:12:32.588 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/00f14d13-c2d7-4e46-a7d3-86627f7ad110 2024-04-26T20:12:32.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:32.589 DEBUG:teuthology.orchestra.run.smithi094:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/00f14d13-c2d7-4e46-a7d3-86627f7ad110 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-04-26T20:12:32.719 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-04-26T20:12:32.719 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-04-26T20:12:32.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:32 smithi094 ceph-mon[28050]: pgmap v456: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:33.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:32 smithi103 ceph-mon[32635]: pgmap v456: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:33.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:32 smithi204 ceph-mon[32618]: pgmap v456: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:33.498 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-04-26T20:12:33.498 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-04-26T20:12:33.498 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.94:3300,172.21.15.103:3300,172.21.15.204:3300" 2024-04-26T20:12:33.498 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with new device syntax: 0@420d0bd6-0407-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/00f14d13-c2d7-4e46-a7d3-86627f7ad110 2024-04-26T20:12:33.498 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.94:3300/172.21.15.103:3300/172.21.15.204:3300" will pass to kernel 2024-04-26T20:12:33.507 INFO:teuthology.orchestra.run.smithi094.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T20:12:33.507 INFO:teuthology.orchestra.run.smithi094.stdout: You just mounted a file system that supports labels which does not 2024-04-26T20:12:33.507 INFO:teuthology.orchestra.run.smithi094.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T20:12:33.507 INFO:teuthology.orchestra.run.smithi094.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T20:12:33.507 INFO:teuthology.orchestra.run.smithi094.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T20:12:33.513 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T20:12:33.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:12:33.513 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T20:12:33.596 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T20:12:33.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:33 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T20:12:33.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:12:33.725 INFO:teuthology.orchestra.run.smithi094.stdout:client25373 2024-04-26T20:12:33.725 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:12:33.726 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:12:33.778 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T20:12:33.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:33 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T20:12:33.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:12:33.940 INFO:teuthology.orchestra.run.smithi094.stdout:client25373 2024-04-26T20:12:33.940 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T20:12:34.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T20:12:34.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:12:34.055 INFO:teuthology.orchestra.run.smithi094.stdout:client25373 2024-04-26T20:12:34.055 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/420d0bd6-0407-11ef-bc93-c7b262605968.client25373/status') 2024-04-26T20:12:34.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:12:34.192 INFO:teuthology.orchestra.run.smithi094.stdout:instance: client.25373 (3)172.21.15.94:0/3149789400 2024-04-26T20:12:34.192 INFO:teuthology.orchestra.run.smithi094.stdout:blocklisted: false 2024-04-26T20:12:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records in 2024-04-26T20:12:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records out 2024-04-26T20:12:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:71 bytes copied, 9.6868e-05 s, 733 kB/s 2024-04-26T20:12:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:12:34.205 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T20:12:34.214 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T20:12:34.214 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-04-26T20:12:34.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-26T20:12:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.415+0000 7fe800a00640 1 -- 172.21.15.94:0/2623595324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078ed0 msgr2=0x7fe7fc079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.415+0000 7fe800a00640 1 --2- 172.21.15.94:0/2623595324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078ed0 0x7fe7fc079770 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009ca0 tx=0x7fe7f402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.415+0000 7fe800a00640 1 -- 172.21.15.94:0/2623595324 shutdown_connections 2024-04-26T20:12:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.415+0000 7fe800a00640 1 --2- 172.21.15.94:0/2623595324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078ed0 0x7fe7fc079770 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.415+0000 7fe800a00640 1 --2- 172.21.15.94:0/2623595324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7fc078530 0x7fe7fc078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.415+0000 7fe800a00640 1 --2- 172.21.15.94:0/2623595324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7fc0772f0 0x7fe7fc077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.415+0000 7fe800a00640 1 -- 172.21.15.94:0/2623595324 >> 172.21.15.94:0/2623595324 conn(0x7fe7fc100150 msgr2=0x7fe7fc102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:34.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.415+0000 7fe800a00640 1 -- 172.21.15.94:0/2623595324 shutdown_connections 2024-04-26T20:12:34.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.415+0000 7fe800a00640 1 -- 172.21.15.94:0/2623595324 wait complete. 2024-04-26T20:12:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.416+0000 7fe800a00640 1 Processor -- start 2024-04-26T20:12:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.416+0000 7fe800a00640 1 -- start start 2024-04-26T20:12:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe800a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7fc0772f0 0x7fe7fc19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe800a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078530 0x7fe7fc19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe800a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7fc078ed0 0x7fe7fc1a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe800a00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc114b40 con 0x7fe7fc078530 2024-04-26T20:12:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe800a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc1149c0 con 0x7fe7fc078ed0 2024-04-26T20:12:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe800a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc114cc0 con 0x7fe7fc0772f0 2024-04-26T20:12:34.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe7f1400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078530 0x7fe7fc19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe7f1400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078530 0x7fe7fc19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57208/0 (socket says 172.21.15.94:57208) 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe7f1400640 1 -- 172.21.15.94:0/4001742293 learned_addr learned my addr 172.21.15.94:0/4001742293 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe7f9e00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7fc0772f0 0x7fe7fc19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe7fa800640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7fc078ed0 0x7fe7fc1a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe7f1400640 1 -- 172.21.15.94:0/4001742293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7fc0772f0 msgr2=0x7fe7fc19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.417+0000 7fe7f1400640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7fc0772f0 0x7fe7fc19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe7f1400640 1 -- 172.21.15.94:0/4001742293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7fc078ed0 msgr2=0x7fe7fc1a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe7f1400640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7fc078ed0 0x7fe7fc1a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe7f1400640 1 -- 172.21.15.94:0/4001742293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7fc1a6cf0 con 0x7fe7fc078530 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe7f9e00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7fc0772f0 0x7fe7fc19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe7f1400640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078530 0x7fe7fc19f610 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400e9f0 tx=0x7fe7e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe7f3400640 1 -- 172.21.15.94:0/4001742293 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e400cd80 con 0x7fe7fc078530 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe7f3400640 1 -- 172.21.15.94:0/4001742293 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7e40108e0 con 0x7fe7fc078530 2024-04-26T20:12:34.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe7f3400640 1 -- 172.21.15.94:0/4001742293 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4018980 con 0x7fe7fc078530 2024-04-26T20:12:34.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7fc1a6fe0 con 0x7fe7fc078530 2024-04-26T20:12:34.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.418+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7fc1a74f0 con 0x7fe7fc078530 2024-04-26T20:12:34.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.420+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7fc111530 con 0x7fe7fc078530 2024-04-26T20:12:34.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.422+0000 7fe7f3400640 1 -- 172.21.15.94:0/4001742293 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe7e4010430 con 0x7fe7fc078530 2024-04-26T20:12:34.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.423+0000 7fe7f3400640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7e00a3d70 0x7fe7e00a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:34.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.423+0000 7fe7f3400640 1 -- 172.21.15.94:0/4001742293 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fe7e40d0c20 con 0x7fe7fc078530 2024-04-26T20:12:34.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.423+0000 7fe7f9e00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7e00a3d70 0x7fe7e00a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:34.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.424+0000 7fe7f9e00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7e00a3d70 0x7fe7e00a6230 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009fc0 tx=0x7fe7ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:34.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.428+0000 7fe7f3400640 1 -- 172.21.15.94:0/4001742293 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe7e409ca20 con 0x7fe7fc078530 2024-04-26T20:12:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.669+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fe7fc02dd20 con 0x7fe7fc078530 2024-04-26T20:12:34.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.735+0000 7fe7f3400640 1 -- 172.21.15.94:0/4001742293 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7fe7e40a0d20 con 0x7fe7fc078530 2024-04-26T20:12:34.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7e00a3d70 msgr2=0x7fe7e00a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:34.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7e00a3d70 0x7fe7e00a6230 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009fc0 tx=0x7fe7ec005f90 comp rx=0 tx=0).stop 2024-04-26T20:12:34.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078530 msgr2=0x7fe7fc19f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:34.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078530 0x7fe7fc19f610 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400e9f0 tx=0x7fe7e400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 shutdown_connections 2024-04-26T20:12:34.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fe7e00a3d70 0x7fe7e00a6230 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7fc078ed0 0x7fe7fc1a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7fc078530 0x7fe7fc19f610 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 --2- 172.21.15.94:0/4001742293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7fc0772f0 0x7fe7fc19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 >> 172.21.15.94:0/4001742293 conn(0x7fe7fc100150 msgr2=0x7fe7fc101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:34.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.740+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 shutdown_connections 2024-04-26T20:12:34.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.741+0000 7fe800a00640 1 -- 172.21.15.94:0/4001742293 wait complete. 2024-04-26T20:12:34.762 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-26T20:12:34.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.962+0000 7f325fe00640 1 -- 172.21.15.94:0/483377452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3260108de0 msgr2=0x7f3260109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:34.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.962+0000 7f325fe00640 1 --2- 172.21.15.94:0/483377452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3260108de0 0x7f3260109200 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3250009900 tx=0x7f325002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.962+0000 7f325fe00640 1 -- 172.21.15.94:0/483377452 shutdown_connections 2024-04-26T20:12:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.962+0000 7f325fe00640 1 --2- 172.21.15.94:0/483377452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f326010a990 0x7f3260111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.962+0000 7f325fe00640 1 --2- 172.21.15.94:0/483377452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3260109fd0 0x7f326010a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.962+0000 7f325fe00640 1 --2- 172.21.15.94:0/483377452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3260108de0 0x7f3260109200 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.962+0000 7f325fe00640 1 -- 172.21.15.94:0/483377452 >> 172.21.15.94:0/483377452 conn(0x7f3260077dc0 msgr2=0x7f326007a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:12:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.962+0000 7f325fe00640 1 -- 172.21.15.94:0/483377452 shutdown_connections 2024-04-26T20:12:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.962+0000 7f325fe00640 1 -- 172.21.15.94:0/483377452 wait complete. 2024-04-26T20:12:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.963+0000 7f325fe00640 1 Processor -- start 2024-04-26T20:12:34.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.963+0000 7f325fe00640 1 -- start start 2024-04-26T20:12:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3260108de0 0x7f326019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3260109fd0 0x7f326019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325fe00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326010a990 0x7f32601a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325fe00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f32601148e0 con 0x7f326010a990 2024-04-26T20:12:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3260114760 con 0x7f3260108de0 2024-04-26T20:12:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3260114a60 con 0x7f3260109fd0 2024-04-26T20:12:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325f400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326010a990 0x7f32601a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325f400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326010a990 0x7f32601a6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:57220/0 (socket says 172.21.15.94:57220) 2024-04-26T20:12:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325f400640 1 -- 172.21.15.94:0/56252003 learned_addr learned my addr 172.21.15.94:0/56252003 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:12:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325e000640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3260109fd0 0x7f326019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325ea00640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3260108de0 0x7f326019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325f400640 1 -- 172.21.15.94:0/56252003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3260109fd0 msgr2=0x7f326019f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325f400640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3260109fd0 0x7f326019f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325f400640 1 -- 172.21.15.94:0/56252003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3260108de0 msgr2=0x7f326019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:12:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325f400640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3260108de0 0x7f326019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:12:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.964+0000 7f325f400640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32601a6c00 con 0x7f326010a990 2024-04-26T20:12:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.965+0000 7f325f400640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326010a990 0x7f32601a6590 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f325800eb10 tx=0x7f325800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.965+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258004b20 con 0x7f326010a990 2024-04-26T20:12:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.965+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3258004cc0 con 0x7f326010a990 2024-04-26T20:12:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.965+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258010670 con 0x7f326010a990 2024-04-26T20:12:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.965+0000 7f325fe00640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32601a6ef0 con 0x7f326010a990 2024-04-26T20:12:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.965+0000 7f325fe00640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32601a73b0 con 0x7f326010a990 2024-04-26T20:12:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.966+0000 7f325fe00640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3260110210 con 0x7f326010a990 2024-04-26T20:12:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.968+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3258005020 con 0x7f326010a990 2024-04-26T20:12:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.970+0000 7f3257400640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440a3c30 0x7f32440a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:12:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.970+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f32580d0ee0 con 0x7f326010a990 2024-04-26T20:12:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.970+0000 7f325ea00640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440a3c30 0x7f32440a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:12:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.971+0000 7f325ea00640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440a3c30 0x7f32440a60f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3250009230 tx=0x7f32500314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:12:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:34.974+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f325809d330 con 0x7f326010a990 2024-04-26T20:12:35.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:34 smithi103 ceph-mon[32635]: pgmap v457: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:35.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:34 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4001742293' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T20:12:35.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:34 smithi204 ceph-mon[32618]: pgmap v457: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:35.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:34 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4001742293' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T20:12:35.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:12:35.205+0000 7f325fe00640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f3260062e10 con 0x7f32440a3c30 2024-04-26T20:12:35.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:34 smithi094 ceph-mon[28050]: pgmap v457: 129 pgs: 129 active+clean; 636 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:35.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:34 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4001742293' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T20:12:36.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:35 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4001742293' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T20:12:36.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:35 smithi103 ceph-mon[32635]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T20:12:36.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:35 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4001742293' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T20:12:36.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:35 smithi204 ceph-mon[32618]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T20:12:36.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:35 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4001742293' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T20:12:36.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:35 smithi094 ceph-mon[28050]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T20:12:37.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:36 smithi103 ceph-mon[32635]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:37.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:36 smithi103 ceph-mon[32635]: pgmap v458: 129 pgs: 129 active+clean; 637 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:12:37.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:36 smithi204 ceph-mon[32618]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:37.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:36 smithi204 ceph-mon[32618]: pgmap v458: 129 pgs: 129 active+clean; 637 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:12:37.240 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:36 smithi094 ceph-mon[28050]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:12:37.240 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:36 smithi094 ceph-mon[28050]: pgmap v458: 129 pgs: 129 active+clean; 637 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-26T20:12:39.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:38 smithi103 ceph-mon[32635]: pgmap v459: 129 pgs: 129 active+clean; 637 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T20:12:39.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:38 smithi204 ceph-mon[32618]: pgmap v459: 129 pgs: 129 active+clean; 637 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T20:12:39.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:38 smithi094 ceph-mon[28050]: pgmap v459: 129 pgs: 129 active+clean; 637 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T20:12:40.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:40 smithi094 ceph-mon[28050]: pgmap v460: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-26T20:12:41.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:40 smithi103 ceph-mon[32635]: pgmap v460: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-26T20:12:41.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:40 smithi204 ceph-mon[32618]: pgmap v460: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-26T20:12:42.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:41 smithi103 ceph-mon[32635]: pgmap v461: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-26T20:12:42.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:41 smithi204 ceph-mon[32618]: pgmap v461: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-26T20:12:42.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:41 smithi094 ceph-mon[28050]: pgmap v461: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-26T20:12:44.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:44 smithi094 ceph-mon[28050]: pgmap v462: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-26T20:12:45.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:44 smithi103 ceph-mon[32635]: pgmap v462: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-26T20:12:45.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:44 smithi204 ceph-mon[32618]: pgmap v462: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-26T20:12:46.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:46 smithi094 ceph-mon[28050]: pgmap v463: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-26T20:12:47.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:46 smithi103 ceph-mon[32635]: pgmap v463: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-26T20:12:47.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:46 smithi204 ceph-mon[32618]: pgmap v463: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-26T20:12:49.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:48 smithi103 ceph-mon[32635]: pgmap v464: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T20:12:49.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:48 smithi204 ceph-mon[32618]: pgmap v464: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T20:12:49.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:48 smithi094 ceph-mon[28050]: pgmap v464: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T20:12:51.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:50 smithi103 ceph-mon[32635]: pgmap v465: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T20:12:51.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:50 smithi204 ceph-mon[32618]: pgmap v465: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T20:12:51.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:50 smithi094 ceph-mon[28050]: pgmap v465: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T20:12:53.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:52 smithi103 ceph-mon[32635]: pgmap v466: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:53.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:52 smithi204 ceph-mon[32618]: pgmap v466: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:53.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:52 smithi094 ceph-mon[28050]: pgmap v466: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:12:55.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:54 smithi103 ceph-mon[32635]: pgmap v467: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:55.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:54 smithi204 ceph-mon[32618]: pgmap v467: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:54 smithi094 ceph-mon[28050]: pgmap v467: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:12:56.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:55 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:12:56.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:55 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:12:56.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:55 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:12:57.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:56 smithi103 ceph-mon[32635]: pgmap v468: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:57.051 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:56 smithi094 ceph-mon[28050]: pgmap v468: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:57.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:56 smithi204 ceph-mon[32618]: pgmap v468: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:12:57.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:12:57.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:12:57.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:12:57.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:12:57 smithi094 ceph-mon[28050]: pgmap v469: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:58.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:12:58.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:12:58.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:12:58.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:12:57 smithi103 ceph-mon[32635]: pgmap v469: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:12:58.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:12:58.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:12:58.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:12:58.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:12:57 smithi204 ceph-mon[32618]: pgmap v469: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:01.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:00 smithi103 ceph-mon[32635]: pgmap v470: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:01.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:00 smithi204 ceph-mon[32618]: pgmap v470: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:01.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:00 smithi094 ceph-mon[28050]: pgmap v470: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:03.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:02 smithi103 ceph-mon[32635]: pgmap v471: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:03.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:02 smithi204 ceph-mon[32618]: pgmap v471: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:03.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:02 smithi094 ceph-mon[28050]: pgmap v471: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:05.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:04 smithi103 ceph-mon[32635]: pgmap v472: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:05.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:04 smithi204 ceph-mon[32618]: pgmap v472: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:05.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:04 smithi094 ceph-mon[28050]: pgmap v472: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:07.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:06 smithi103 ceph-mon[32635]: pgmap v473: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:07.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:06 smithi204 ceph-mon[32618]: pgmap v473: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:07.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:06 smithi094 ceph-mon[28050]: pgmap v473: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:09.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:08 smithi103 ceph-mon[32635]: pgmap v474: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:09.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:08 smithi204 ceph-mon[32618]: pgmap v474: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:09.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:08 smithi094 ceph-mon[28050]: pgmap v474: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:11.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:10 smithi103 ceph-mon[32635]: pgmap v475: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:11.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:10 smithi204 ceph-mon[32618]: pgmap v475: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:11.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:10 smithi094 ceph-mon[28050]: pgmap v475: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:13.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:12 smithi103 ceph-mon[32635]: pgmap v476: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:13.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:12 smithi204 ceph-mon[32618]: pgmap v476: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:13.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:12 smithi094 ceph-mon[28050]: pgmap v476: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:14.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:13 smithi204 ceph-mon[32618]: pgmap v477: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:14.139 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:13 smithi094 ceph-mon[28050]: pgmap v477: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:14.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:13 smithi103 ceph-mon[32635]: pgmap v477: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:16.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:16 smithi094 ceph-mon[28050]: pgmap v478: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:17.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:16 smithi103 ceph-mon[32635]: pgmap v478: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:17.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:16 smithi204 ceph-mon[32618]: pgmap v478: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:19.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:18 smithi103 ceph-mon[32635]: pgmap v479: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:19.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:18 smithi204 ceph-mon[32618]: pgmap v479: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:19.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:18 smithi094 ceph-mon[28050]: pgmap v479: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:21.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:20 smithi103 ceph-mon[32635]: pgmap v480: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:21.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:20 smithi204 ceph-mon[32618]: pgmap v480: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:20 smithi094 ceph-mon[28050]: pgmap v480: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:23.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:22 smithi103 ceph-mon[32635]: pgmap v481: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:23.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:22 smithi204 ceph-mon[32618]: pgmap v481: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:23.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:22 smithi094 ceph-mon[28050]: pgmap v481: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:25.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:24 smithi103 ceph-mon[32635]: pgmap v482: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:25.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:24 smithi204 ceph-mon[32618]: pgmap v482: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:25.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:24 smithi094 ceph-mon[28050]: pgmap v482: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:26.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:25 smithi204 ceph-mon[32618]: pgmap v483: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:26.196 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:25 smithi094 ceph-mon[28050]: pgmap v483: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:26.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:25 smithi103 ceph-mon[32635]: pgmap v483: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:29.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:28 smithi103 ceph-mon[32635]: pgmap v484: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:29.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:28 smithi204 ceph-mon[32618]: pgmap v484: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:29.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:28 smithi094 ceph-mon[28050]: pgmap v484: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:31.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:30 smithi103 ceph-mon[32635]: pgmap v485: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:31.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:30 smithi204 ceph-mon[32618]: pgmap v485: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:31.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:30 smithi094 ceph-mon[28050]: pgmap v485: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:33.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:32 smithi103 ceph-mon[32635]: pgmap v486: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:33.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:32 smithi204 ceph-mon[32618]: pgmap v486: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:33.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:32 smithi094 ceph-mon[28050]: pgmap v486: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:35.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:34 smithi103 ceph-mon[32635]: pgmap v487: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:35.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:34 smithi204 ceph-mon[32618]: pgmap v487: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:35.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:34 smithi094 ceph-mon[28050]: pgmap v487: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:36.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:35 smithi204 ceph-mon[32618]: pgmap v488: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:36.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:35 smithi094 ceph-mon[28050]: pgmap v488: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:36.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:35 smithi103 ceph-mon[32635]: pgmap v488: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:39.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:38 smithi103 ceph-mon[32635]: pgmap v489: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:39.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:38 smithi204 ceph-mon[32618]: pgmap v489: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:39.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:38 smithi094 ceph-mon[28050]: pgmap v489: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:41.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:40 smithi103 ceph-mon[32635]: pgmap v490: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:41.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:40 smithi204 ceph-mon[32618]: pgmap v490: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:41.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:40 smithi094 ceph-mon[28050]: pgmap v490: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:43.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:42 smithi103 ceph-mon[32635]: pgmap v491: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:43.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:42 smithi204 ceph-mon[32618]: pgmap v491: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:42 smithi094 ceph-mon[28050]: pgmap v491: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:44.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:43 smithi204 ceph-mon[32618]: pgmap v492: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:44.139 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:43 smithi094 ceph-mon[28050]: pgmap v492: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:44.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:43 smithi103 ceph-mon[32635]: pgmap v492: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:47.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:46 smithi103 ceph-mon[32635]: pgmap v493: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:47.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:46 smithi204 ceph-mon[32618]: pgmap v493: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:47.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:46 smithi094 ceph-mon[28050]: pgmap v493: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:49.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:48 smithi103 ceph-mon[32635]: pgmap v494: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:49.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:48 smithi204 ceph-mon[32618]: pgmap v494: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:49.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:48 smithi094 ceph-mon[28050]: pgmap v494: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:51.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:50 smithi103 ceph-mon[32635]: pgmap v495: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:51.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:50 smithi204 ceph-mon[32618]: pgmap v495: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:51.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:50 smithi094 ceph-mon[28050]: pgmap v495: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:13:52.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:51 smithi204 ceph-mon[32618]: pgmap v496: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:51 smithi094 ceph-mon[28050]: pgmap v496: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:51 smithi103 ceph-mon[32635]: pgmap v496: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:13:55.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:54 smithi103 ceph-mon[32635]: pgmap v497: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:55.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:54 smithi204 ceph-mon[32618]: pgmap v497: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:54 smithi094 ceph-mon[28050]: pgmap v497: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:13:57.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:56 smithi103 ceph-mon[32635]: pgmap v498: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:57.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:56 smithi204 ceph-mon[32618]: pgmap v498: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:57.236 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:56 smithi094 ceph-mon[28050]: pgmap v498: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:13:58.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:57 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:13:58.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:57 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:13:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:57 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:13:59.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:58 smithi204 ceph-mon[32618]: pgmap v499: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:59.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:58 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:13:59.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:58 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:13:59.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:58 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:13:59.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:58 smithi094 ceph-mon[28050]: pgmap v499: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:59.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:13:59.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:13:59.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:13:59.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:58 smithi103 ceph-mon[32635]: pgmap v499: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:13:59.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:58 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:13:59.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:58 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:13:59.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:58 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:14:00.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:13:59 smithi204 ceph-mon[32618]: pgmap v500: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:00.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:13:59 smithi094 ceph-mon[28050]: pgmap v500: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:00.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:13:59 smithi103 ceph-mon[32635]: pgmap v500: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:03.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:02 smithi103 ceph-mon[32635]: pgmap v501: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:03.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:02 smithi204 ceph-mon[32618]: pgmap v501: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:03.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:02 smithi094 ceph-mon[28050]: pgmap v501: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:05.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:04 smithi103 ceph-mon[32635]: pgmap v502: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:05.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:04 smithi204 ceph-mon[32618]: pgmap v502: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:05.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:04 smithi094 ceph-mon[28050]: pgmap v502: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:06.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:05 smithi204 ceph-mon[32618]: pgmap v503: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:05 smithi094 ceph-mon[28050]: pgmap v503: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:06.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:05 smithi103 ceph-mon[32635]: pgmap v503: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:09.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:08 smithi103 ceph-mon[32635]: pgmap v504: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:14:09.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:08 smithi204 ceph-mon[32618]: pgmap v504: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:14:09.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:08 smithi094 ceph-mon[28050]: pgmap v504: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:14:11.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:10 smithi204 ceph-mon[32618]: pgmap v505: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:14:11.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:10 smithi094 ceph-mon[28050]: pgmap v505: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:14:11.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:10 smithi103 ceph-mon[32635]: pgmap v505: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:14:12.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:11 smithi204 ceph-mon[32618]: pgmap v506: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:14:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:11 smithi094 ceph-mon[28050]: pgmap v506: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:14:12.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:11 smithi103 ceph-mon[32635]: pgmap v506: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:14:15.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:14 smithi103 ceph-mon[32635]: pgmap v507: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:14:15.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:14 smithi204 ceph-mon[32618]: pgmap v507: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:14:15.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:14 smithi094 ceph-mon[28050]: pgmap v507: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:14:16.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:15 smithi204 ceph-mon[32618]: pgmap v508: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:14:16.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:15 smithi094 ceph-mon[28050]: pgmap v508: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:14:16.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:15 smithi103 ceph-mon[32635]: pgmap v508: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:14:19.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:18 smithi204 ceph-mon[32618]: pgmap v509: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:14:19.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:18 smithi094 ceph-mon[28050]: pgmap v509: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:14:19.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:18 smithi103 ceph-mon[32635]: pgmap v509: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T20:14:20.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:19 smithi204 ceph-mon[32618]: pgmap v510: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:20.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:19 smithi094 ceph-mon[28050]: pgmap v510: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:20.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:19 smithi103 ceph-mon[32635]: pgmap v510: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:23.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:22 smithi204 ceph-mon[32618]: pgmap v511: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:23.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:22 smithi094 ceph-mon[28050]: pgmap v511: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:23.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:22 smithi103 ceph-mon[32635]: pgmap v511: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:24.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:23 smithi204 ceph-mon[32618]: pgmap v512: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:24.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:23 smithi094 ceph-mon[28050]: pgmap v512: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:23 smithi103 ceph-mon[32635]: pgmap v512: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:25 smithi204 ceph-mon[32618]: pgmap v513: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:26.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:25 smithi094 ceph-mon[28050]: pgmap v513: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:26.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:25 smithi103 ceph-mon[32635]: pgmap v513: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:28.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:27 smithi204 ceph-mon[32618]: pgmap v514: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:27 smithi094 ceph-mon[28050]: pgmap v514: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:27 smithi103 ceph-mon[32635]: pgmap v514: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:30.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:29 smithi204 ceph-mon[32618]: pgmap v515: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:30.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:29 smithi094 ceph-mon[28050]: pgmap v515: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:30.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:29 smithi103 ceph-mon[32635]: pgmap v515: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:32.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:31 smithi204 ceph-mon[32618]: pgmap v516: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:32.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:31 smithi094 ceph-mon[28050]: pgmap v516: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:32.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:31 smithi103 ceph-mon[32635]: pgmap v516: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:34.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:33 smithi204 ceph-mon[32618]: pgmap v517: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:34.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:33 smithi094 ceph-mon[28050]: pgmap v517: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:34.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:33 smithi103 ceph-mon[32635]: pgmap v517: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:36.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:35 smithi204 ceph-mon[32618]: pgmap v518: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:36.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:35 smithi094 ceph-mon[28050]: pgmap v518: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:36.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:35 smithi103 ceph-mon[32635]: pgmap v518: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:38.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:37 smithi204 ceph-mon[32618]: pgmap v519: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:38.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:37 smithi094 ceph-mon[28050]: pgmap v519: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:38.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:37 smithi103 ceph-mon[32635]: pgmap v519: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:40.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:39 smithi204 ceph-mon[32618]: pgmap v520: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:39 smithi094 ceph-mon[28050]: pgmap v520: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:40.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:39 smithi103 ceph-mon[32635]: pgmap v520: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:42.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:41 smithi204 ceph-mon[32618]: pgmap v521: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:42.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:41 smithi094 ceph-mon[28050]: pgmap v521: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:42.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:41 smithi103 ceph-mon[32635]: pgmap v521: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:44.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:43 smithi204 ceph-mon[32618]: pgmap v522: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:44.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:43 smithi094 ceph-mon[28050]: pgmap v522: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:44.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:43 smithi103 ceph-mon[32635]: pgmap v522: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:46.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:45 smithi204 ceph-mon[32618]: pgmap v523: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:46.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:45 smithi094 ceph-mon[28050]: pgmap v523: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:46.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:45 smithi103 ceph-mon[32635]: pgmap v523: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:48.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:47 smithi204 ceph-mon[32618]: pgmap v524: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:48.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:47 smithi094 ceph-mon[28050]: pgmap v524: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:48.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:47 smithi103 ceph-mon[32635]: pgmap v524: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:50.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:49 smithi204 ceph-mon[32618]: pgmap v525: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:49 smithi094 ceph-mon[28050]: pgmap v525: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:50.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:49 smithi103 ceph-mon[32635]: pgmap v525: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:14:52.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:51 smithi204 ceph-mon[32618]: pgmap v526: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:51 smithi094 ceph-mon[28050]: pgmap v526: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:51 smithi103 ceph-mon[32635]: pgmap v526: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:14:54.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:53 smithi204 ceph-mon[32618]: pgmap v527: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:54.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:53 smithi094 ceph-mon[28050]: pgmap v527: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:54.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:53 smithi103 ceph-mon[32635]: pgmap v527: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:14:56.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:55 smithi204 ceph-mon[32618]: pgmap v528: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:56.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:55 smithi094 ceph-mon[28050]: pgmap v528: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:56.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:55 smithi103 ceph-mon[32635]: pgmap v528: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:14:58.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:57 smithi204 ceph-mon[32618]: pgmap v529: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:57 smithi094 ceph-mon[28050]: pgmap v529: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:57 smithi103 ceph-mon[32635]: pgmap v529: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:14:59.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:14:58 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:14:59.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:14:58 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:14:59.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:14:58 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:15:00.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:00 smithi094 ceph-mon[28050]: pgmap v530: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:00.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:00 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:15:00.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:00 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:15:00.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:00 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:15:01.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:00 smithi103 ceph-mon[32635]: pgmap v530: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:01.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:00 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:15:01.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:00 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:15:01.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:00 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:15:01.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:00 smithi204 ceph-mon[32618]: pgmap v530: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:01.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:00 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:15:01.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:00 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:15:01.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:00 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:15:02.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:01 smithi204 ceph-mon[32618]: pgmap v531: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:01 smithi094 ceph-mon[28050]: pgmap v531: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:01 smithi103 ceph-mon[32635]: pgmap v531: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:04.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:03 smithi204 ceph-mon[32618]: pgmap v532: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:04.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:03 smithi094 ceph-mon[28050]: pgmap v532: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:03 smithi103 ceph-mon[32635]: pgmap v532: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:06.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:05 smithi204 ceph-mon[32618]: pgmap v533: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:05 smithi094 ceph-mon[28050]: pgmap v533: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:06.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:05 smithi103 ceph-mon[32635]: pgmap v533: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:08.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:07 smithi204 ceph-mon[32618]: pgmap v534: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:08.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:07 smithi094 ceph-mon[28050]: pgmap v534: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:08.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:07 smithi103 ceph-mon[32635]: pgmap v534: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:10.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:09 smithi204 ceph-mon[32618]: pgmap v535: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:10.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:09 smithi094 ceph-mon[28050]: pgmap v535: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:10.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:09 smithi103 ceph-mon[32635]: pgmap v535: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:12.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:11 smithi204 ceph-mon[32618]: pgmap v536: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:11 smithi094 ceph-mon[28050]: pgmap v536: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:12.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:11 smithi103 ceph-mon[32635]: pgmap v536: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:14.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:13 smithi204 ceph-mon[32618]: pgmap v537: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:14.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:13 smithi094 ceph-mon[28050]: pgmap v537: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:14.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:13 smithi103 ceph-mon[32635]: pgmap v537: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:16.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:15 smithi204 ceph-mon[32618]: pgmap v538: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:16.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:15 smithi094 ceph-mon[28050]: pgmap v538: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:16.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:15 smithi103 ceph-mon[32635]: pgmap v538: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:18.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:17 smithi204 ceph-mon[32618]: pgmap v539: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:18.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:17 smithi094 ceph-mon[28050]: pgmap v539: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:18.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:17 smithi103 ceph-mon[32635]: pgmap v539: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:20.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:19 smithi204 ceph-mon[32618]: pgmap v540: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:20.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:19 smithi094 ceph-mon[28050]: pgmap v540: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:20.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:19 smithi103 ceph-mon[32635]: pgmap v540: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:22.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:21 smithi204 ceph-mon[32618]: pgmap v541: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:22.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:21 smithi094 ceph-mon[28050]: pgmap v541: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:22.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:21 smithi103 ceph-mon[32635]: pgmap v541: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:24.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:23 smithi204 ceph-mon[32618]: pgmap v542: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:24.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:23 smithi094 ceph-mon[28050]: pgmap v542: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:23 smithi103 ceph-mon[32635]: pgmap v542: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:26.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:25 smithi204 ceph-mon[32618]: pgmap v543: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:26.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:25 smithi094 ceph-mon[28050]: pgmap v543: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:26.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:25 smithi103 ceph-mon[32635]: pgmap v543: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:28.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:27 smithi204 ceph-mon[32618]: pgmap v544: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:27 smithi094 ceph-mon[28050]: pgmap v544: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:27 smithi103 ceph-mon[32635]: pgmap v544: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:30.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:29 smithi204 ceph-mon[32618]: pgmap v545: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:30.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:29 smithi094 ceph-mon[28050]: pgmap v545: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:30.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:29 smithi103 ceph-mon[32635]: pgmap v545: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:32.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:31 smithi204 ceph-mon[32618]: pgmap v546: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:32.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:31 smithi094 ceph-mon[28050]: pgmap v546: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:32.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:31 smithi103 ceph-mon[32635]: pgmap v546: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:34.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:33 smithi204 ceph-mon[32618]: pgmap v547: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:34.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:33 smithi094 ceph-mon[28050]: pgmap v547: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:34.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:33 smithi103 ceph-mon[32635]: pgmap v547: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:34.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:15:34.993+0000 7f3256000640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f32380034c0 con 0x7f326010a990 2024-04-26T20:15:34.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:15:34.994+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f32380034c0 con 0x7f326010a990 2024-04-26T20:15:36.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:35 smithi204 ceph-mon[32618]: pgmap v548: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:36.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:35 smithi094 ceph-mon[28050]: pgmap v548: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:36.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:35 smithi103 ceph-mon[32635]: pgmap v548: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:38.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:37 smithi204 ceph-mon[32618]: pgmap v549: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:38.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:37 smithi094 ceph-mon[28050]: pgmap v549: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:38.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:37 smithi103 ceph-mon[32635]: pgmap v549: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:40.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:39 smithi204 ceph-mon[32618]: pgmap v550: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:39 smithi094 ceph-mon[28050]: pgmap v550: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:40.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:39 smithi103 ceph-mon[32635]: pgmap v550: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:42.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:41 smithi204 ceph-mon[32618]: pgmap v551: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:42.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:41 smithi094 ceph-mon[28050]: pgmap v551: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:42.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:41 smithi103 ceph-mon[32635]: pgmap v551: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:44.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:43 smithi204 ceph-mon[32618]: pgmap v552: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:44.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:43 smithi094 ceph-mon[28050]: pgmap v552: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:44.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:43 smithi103 ceph-mon[32635]: pgmap v552: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:46.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:45 smithi204 ceph-mon[32618]: pgmap v553: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:46.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:45 smithi094 ceph-mon[28050]: pgmap v553: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:46.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:45 smithi103 ceph-mon[32635]: pgmap v553: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:48.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:47 smithi204 ceph-mon[32618]: pgmap v554: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:48.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:47 smithi094 ceph-mon[28050]: pgmap v554: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:48.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:47 smithi103 ceph-mon[32635]: pgmap v554: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:50.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:49 smithi204 ceph-mon[32618]: pgmap v555: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:49 smithi094 ceph-mon[28050]: pgmap v555: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:50.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:49 smithi103 ceph-mon[32635]: pgmap v555: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:15:52.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:51 smithi204 ceph-mon[32618]: pgmap v556: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:51 smithi094 ceph-mon[28050]: pgmap v556: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:51 smithi103 ceph-mon[32635]: pgmap v556: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:15:54.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:53 smithi204 ceph-mon[32618]: pgmap v557: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:54.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:53 smithi094 ceph-mon[28050]: pgmap v557: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:54.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:53 smithi103 ceph-mon[32635]: pgmap v557: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:15:56.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:55 smithi204 ceph-mon[32618]: pgmap v558: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:56.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:55 smithi094 ceph-mon[28050]: pgmap v558: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:56.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:55 smithi103 ceph-mon[32635]: pgmap v558: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:15:58.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:57 smithi204 ceph-mon[32618]: pgmap v559: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:57 smithi094 ceph-mon[28050]: pgmap v559: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:15:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:57 smithi103 ceph-mon[32635]: pgmap v559: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:00.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:15:59 smithi204 ceph-mon[32618]: pgmap v560: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:00.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:15:59 smithi094 ceph-mon[28050]: pgmap v560: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:00.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:15:59 smithi103 ceph-mon[32635]: pgmap v560: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:01.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:00 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:16:01.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:00 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:16:01.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:00 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:16:02.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:01 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:16:02.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:01 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:16:02.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:01 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:16:02.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:01 smithi204 ceph-mon[32618]: pgmap v561: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:16:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:16:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:01 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:16:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:01 smithi094 ceph-mon[28050]: pgmap v561: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:01 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:16:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:01 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:16:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:01 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:16:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:01 smithi103 ceph-mon[32635]: pgmap v561: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:04.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:03 smithi204 ceph-mon[32618]: pgmap v562: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:04.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:03 smithi094 ceph-mon[28050]: pgmap v562: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:03 smithi103 ceph-mon[32635]: pgmap v562: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:06.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:05 smithi204 ceph-mon[32618]: pgmap v563: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:05 smithi094 ceph-mon[28050]: pgmap v563: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:06.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:05 smithi103 ceph-mon[32635]: pgmap v563: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:08.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:07 smithi204 ceph-mon[32618]: pgmap v564: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:08.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:07 smithi094 ceph-mon[28050]: pgmap v564: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:08.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:07 smithi103 ceph-mon[32635]: pgmap v564: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:10.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:09 smithi204 ceph-mon[32618]: pgmap v565: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:10.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:09 smithi094 ceph-mon[28050]: pgmap v565: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:10.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:09 smithi103 ceph-mon[32635]: pgmap v565: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:12.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:11 smithi204 ceph-mon[32618]: pgmap v566: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:11 smithi094 ceph-mon[28050]: pgmap v566: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:12.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:11 smithi103 ceph-mon[32635]: pgmap v566: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:14.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:13 smithi204 ceph-mon[32618]: pgmap v567: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:14.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:13 smithi094 ceph-mon[28050]: pgmap v567: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:14.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:13 smithi103 ceph-mon[32635]: pgmap v567: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:16.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:15 smithi204 ceph-mon[32618]: pgmap v568: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:16.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:15 smithi094 ceph-mon[28050]: pgmap v568: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:16.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:15 smithi103 ceph-mon[32635]: pgmap v568: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:18.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:17 smithi204 ceph-mon[32618]: pgmap v569: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:18.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:17 smithi094 ceph-mon[28050]: pgmap v569: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:18.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:17 smithi103 ceph-mon[32635]: pgmap v569: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:20.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:19 smithi204 ceph-mon[32618]: pgmap v570: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:20.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:19 smithi094 ceph-mon[28050]: pgmap v570: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:20.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:19 smithi103 ceph-mon[32635]: pgmap v570: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:22.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:21 smithi204 ceph-mon[32618]: pgmap v571: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:22.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:21 smithi094 ceph-mon[28050]: pgmap v571: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:22.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:21 smithi103 ceph-mon[32635]: pgmap v571: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:24.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:23 smithi204 ceph-mon[32618]: pgmap v572: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:24.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:23 smithi094 ceph-mon[28050]: pgmap v572: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:23 smithi103 ceph-mon[32635]: pgmap v572: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:26.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:25 smithi204 ceph-mon[32618]: pgmap v573: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:26.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:25 smithi094 ceph-mon[28050]: pgmap v573: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:26.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:25 smithi103 ceph-mon[32635]: pgmap v573: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:28.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:27 smithi204 ceph-mon[32618]: pgmap v574: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:27 smithi094 ceph-mon[28050]: pgmap v574: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:27 smithi103 ceph-mon[32635]: pgmap v574: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:30.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:29 smithi204 ceph-mon[32618]: pgmap v575: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:30.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:29 smithi094 ceph-mon[28050]: pgmap v575: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:30.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:29 smithi103 ceph-mon[32635]: pgmap v575: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:32.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:31 smithi204 ceph-mon[32618]: pgmap v576: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:32.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:31 smithi094 ceph-mon[28050]: pgmap v576: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:32.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:31 smithi103 ceph-mon[32635]: pgmap v576: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:34.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:33 smithi204 ceph-mon[32618]: pgmap v577: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:34.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:33 smithi094 ceph-mon[28050]: pgmap v577: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:34.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:33 smithi103 ceph-mon[32635]: pgmap v577: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:36.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:35 smithi204 ceph-mon[32618]: pgmap v578: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:36.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:35 smithi094 ceph-mon[28050]: pgmap v578: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:36.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:35 smithi103 ceph-mon[32635]: pgmap v578: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:38.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:37 smithi204 ceph-mon[32618]: pgmap v579: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:38.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:37 smithi094 ceph-mon[28050]: pgmap v579: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:38.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:37 smithi103 ceph-mon[32635]: pgmap v579: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:39 smithi204 ceph-mon[32618]: pgmap v580: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:39 smithi094 ceph-mon[28050]: pgmap v580: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:40.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:39 smithi103 ceph-mon[32635]: pgmap v580: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:42.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:41 smithi204 ceph-mon[32618]: pgmap v581: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:42.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:41 smithi094 ceph-mon[28050]: pgmap v581: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:42.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:41 smithi103 ceph-mon[32635]: pgmap v581: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:44.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:43 smithi204 ceph-mon[32618]: pgmap v582: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:44.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:43 smithi094 ceph-mon[28050]: pgmap v582: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:44.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:43 smithi103 ceph-mon[32635]: pgmap v582: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:46.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:45 smithi204 ceph-mon[32618]: pgmap v583: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:46.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:45 smithi094 ceph-mon[28050]: pgmap v583: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:46.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:45 smithi103 ceph-mon[32635]: pgmap v583: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:48.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:47 smithi204 ceph-mon[32618]: pgmap v584: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:48.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:47 smithi094 ceph-mon[28050]: pgmap v584: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:48.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:47 smithi103 ceph-mon[32635]: pgmap v584: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:50.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:49 smithi204 ceph-mon[32618]: pgmap v585: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:49 smithi094 ceph-mon[28050]: pgmap v585: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:50.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:49 smithi103 ceph-mon[32635]: pgmap v585: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T20:16:52.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:51 smithi204 ceph-mon[32618]: pgmap v586: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:51 smithi094 ceph-mon[28050]: pgmap v586: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:51 smithi103 ceph-mon[32635]: pgmap v586: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:16:54.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:53 smithi204 ceph-mon[32618]: pgmap v587: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:54.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:53 smithi094 ceph-mon[28050]: pgmap v587: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:54.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:53 smithi103 ceph-mon[32635]: pgmap v587: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:16:56.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:55 smithi204 ceph-mon[32618]: pgmap v588: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:56.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:55 smithi094 ceph-mon[28050]: pgmap v588: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:56.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:55 smithi103 ceph-mon[32635]: pgmap v588: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:16:58.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:57 smithi204 ceph-mon[32618]: pgmap v589: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:57 smithi094 ceph-mon[28050]: pgmap v589: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:16:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:57 smithi103 ceph-mon[32635]: pgmap v589: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:00.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:16:59 smithi204 ceph-mon[32618]: pgmap v590: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:00.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:16:59 smithi094 ceph-mon[28050]: pgmap v590: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:00.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:16:59 smithi103 ceph-mon[32635]: pgmap v590: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:02.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:01 smithi204 ceph-mon[32618]: pgmap v591: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:01 smithi094 ceph-mon[28050]: pgmap v591: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:01 smithi103 ceph-mon[32635]: pgmap v591: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:03.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:17:03.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:02 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:02 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-04-26T20:17:03.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:02 smithi094 ceph-mon[28050]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-04-26T20:17:03.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:17:03.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:02 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:02 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-04-26T20:17:03.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:02 smithi103 ceph-mon[32635]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-04-26T20:17:03.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:17:03.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:02 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:03.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:02 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-04-26T20:17:03.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:02 smithi204 ceph-mon[32618]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-04-26T20:17:04.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi094 to 176.4M 2024-04-26T20:17:04.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: Unable to set osd_memory_target on smithi094 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-04-26T20:17:04.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:17:04.238 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:17:04.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:17:04.239 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:03 smithi094 ceph-mon[28050]: pgmap v592: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi094 to 176.4M 2024-04-26T20:17:04.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: Unable to set osd_memory_target on smithi094 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-04-26T20:17:04.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:17:04.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:17:04.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:17:04.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:03 smithi103 ceph-mon[32635]: pgmap v592: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:04.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:17:04.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi094 to 176.4M 2024-04-26T20:17:04.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: Unable to set osd_memory_target on smithi094 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-04-26T20:17:04.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:17:04.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:17:04.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:17:04.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:03 smithi204 ceph-mon[32618]: pgmap v592: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:05 smithi094 ceph-mon[28050]: pgmap v593: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:06.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:05 smithi103 ceph-mon[32635]: pgmap v593: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:06.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:05 smithi204 ceph-mon[32618]: pgmap v593: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:09.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:08 smithi204 ceph-mon[32618]: pgmap v594: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:09.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:08 smithi094 ceph-mon[28050]: pgmap v594: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:09.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:08 smithi103 ceph-mon[32635]: pgmap v594: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:11.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:10 smithi094 ceph-mon[28050]: pgmap v595: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:11.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:10 smithi103 ceph-mon[32635]: pgmap v595: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:11.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:10 smithi204 ceph-mon[32618]: pgmap v595: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:13.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:12 smithi094 ceph-mon[28050]: pgmap v596: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:13.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:12 smithi103 ceph-mon[32635]: pgmap v596: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:13.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:12 smithi204 ceph-mon[32618]: pgmap v596: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:15.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:14 smithi094 ceph-mon[28050]: pgmap v597: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:15.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:14 smithi103 ceph-mon[32635]: pgmap v597: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:15.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:14 smithi204 ceph-mon[32618]: pgmap v597: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:17.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:16 smithi094 ceph-mon[28050]: pgmap v598: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:17.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:16 smithi103 ceph-mon[32635]: pgmap v598: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:17.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:16 smithi204 ceph-mon[32618]: pgmap v598: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:19.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:18 smithi094 ceph-mon[28050]: pgmap v599: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:19.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:18 smithi103 ceph-mon[32635]: pgmap v599: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:19.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:18 smithi204 ceph-mon[32618]: pgmap v599: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:20 smithi094 ceph-mon[28050]: pgmap v600: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:21.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:20 smithi103 ceph-mon[32635]: pgmap v600: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:21.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:20 smithi204 ceph-mon[32618]: pgmap v600: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:23.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:22 smithi094 ceph-mon[28050]: pgmap v601: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:23.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:22 smithi103 ceph-mon[32635]: pgmap v601: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:23.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:22 smithi204 ceph-mon[32618]: pgmap v601: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:25 smithi094 ceph-mon[28050]: pgmap v602: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:26.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:25 smithi103 ceph-mon[32635]: pgmap v602: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:26.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:25 smithi204 ceph-mon[32618]: pgmap v602: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:27.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:27 smithi094 ceph-mon[28050]: pgmap v603: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:27 smithi103 ceph-mon[32635]: pgmap v603: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:28.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:27 smithi204 ceph-mon[32618]: pgmap v603: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:29.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:29 smithi094 ceph-mon[28050]: pgmap v604: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:30.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:29 smithi103 ceph-mon[32635]: pgmap v604: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:30.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:29 smithi204 ceph-mon[32618]: pgmap v604: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:31.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:31 smithi094 ceph-mon[28050]: pgmap v605: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:32.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:31 smithi103 ceph-mon[32635]: pgmap v605: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:32.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:31 smithi204 ceph-mon[32618]: pgmap v605: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:34.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:33 smithi103 ceph-mon[32635]: pgmap v606: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:34.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:33 smithi204 ceph-mon[32618]: pgmap v606: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:34.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:33 smithi094 ceph-mon[28050]: pgmap v606: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:36.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:35 smithi103 ceph-mon[32635]: pgmap v607: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:36.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:35 smithi204 ceph-mon[32618]: pgmap v607: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:36.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:35 smithi094 ceph-mon[28050]: pgmap v607: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:37 smithi103 ceph-mon[32635]: pgmap v608: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:38.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:37 smithi204 ceph-mon[32618]: pgmap v608: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:38.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:37 smithi094 ceph-mon[28050]: pgmap v608: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:40.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:39 smithi103 ceph-mon[32635]: pgmap v609: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:40.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:39 smithi204 ceph-mon[32618]: pgmap v609: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:39 smithi094 ceph-mon[28050]: pgmap v609: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:42.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:41 smithi103 ceph-mon[32635]: pgmap v610: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:42.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:41 smithi204 ceph-mon[32618]: pgmap v610: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:42.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:41 smithi094 ceph-mon[28050]: pgmap v610: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:44.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:43 smithi103 ceph-mon[32635]: pgmap v611: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:44.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:43 smithi204 ceph-mon[32618]: pgmap v611: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:44.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:43 smithi094 ceph-mon[28050]: pgmap v611: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:46.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:45 smithi103 ceph-mon[32635]: pgmap v612: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:46.040 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:45 smithi094 ceph-mon[28050]: pgmap v612: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:46.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:45 smithi204 ceph-mon[32618]: pgmap v612: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:48.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:47 smithi103 ceph-mon[32635]: pgmap v613: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:48.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:47 smithi204 ceph-mon[32618]: pgmap v613: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:48.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:47 smithi094 ceph-mon[28050]: pgmap v613: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:50.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:49 smithi103 ceph-mon[32635]: pgmap v614: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:50.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:49 smithi204 ceph-mon[32618]: pgmap v614: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:49 smithi094 ceph-mon[28050]: pgmap v614: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:17:52.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:51 smithi204 ceph-mon[32618]: pgmap v615: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:51 smithi094 ceph-mon[28050]: pgmap v615: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:51 smithi103 ceph-mon[32635]: pgmap v615: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:54.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:53 smithi204 ceph-mon[32618]: pgmap v616: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:54.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:53 smithi094 ceph-mon[28050]: pgmap v616: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:54.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:53 smithi103 ceph-mon[32635]: pgmap v616: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:17:56.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:55 smithi204 ceph-mon[32618]: pgmap v617: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:56.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:55 smithi094 ceph-mon[28050]: pgmap v617: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:56.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:55 smithi103 ceph-mon[32635]: pgmap v617: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:17:58.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:57 smithi204 ceph-mon[32618]: pgmap v618: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:58.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:57 smithi094 ceph-mon[28050]: pgmap v618: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:17:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:57 smithi103 ceph-mon[32635]: pgmap v618: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:00.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:17:59 smithi204 ceph-mon[32618]: pgmap v619: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:00.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:17:59 smithi094 ceph-mon[28050]: pgmap v619: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:00.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:17:59 smithi103 ceph-mon[32635]: pgmap v619: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:02.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:01 smithi204 ceph-mon[32618]: pgmap v620: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:01 smithi094 ceph-mon[28050]: pgmap v620: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:01 smithi103 ceph-mon[32635]: pgmap v620: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:04.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:03 smithi204 ceph-mon[32618]: pgmap v621: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:04.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:03 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:18:04.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:03 smithi094 ceph-mon[28050]: pgmap v621: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:04.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:03 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:18:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:03 smithi103 ceph-mon[32635]: pgmap v621: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:03 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:18:05.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:04 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:04 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:05.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:04 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:18:06.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:05 smithi204 ceph-mon[32618]: pgmap v622: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:06.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:05 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi204 to 176.1M 2024-04-26T20:18:06.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:05 smithi204 ceph-mon[32618]: Unable to set osd_memory_target on smithi204 to 184758067: error parsing value: Value '184758067' is below minimum 939524096 2024-04-26T20:18:06.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:18:06.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:18:06.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:18:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:05 smithi094 ceph-mon[28050]: pgmap v622: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:05 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi204 to 176.1M 2024-04-26T20:18:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:05 smithi094 ceph-mon[28050]: Unable to set osd_memory_target on smithi204 to 184758067: error parsing value: Value '184758067' is below minimum 939524096 2024-04-26T20:18:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:18:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:18:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:18:06.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:05 smithi103 ceph-mon[32635]: pgmap v622: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:06.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:05 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi204 to 176.1M 2024-04-26T20:18:06.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:05 smithi103 ceph-mon[32635]: Unable to set osd_memory_target on smithi204 to 184758067: error parsing value: Value '184758067' is below minimum 939524096 2024-04-26T20:18:06.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:18:06.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:18:06.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:18:08.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:07 smithi204 ceph-mon[32618]: pgmap v623: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:08.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:07 smithi094 ceph-mon[28050]: pgmap v623: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:08.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:07 smithi103 ceph-mon[32635]: pgmap v623: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:09.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:08 smithi094 ceph-mon[28050]: pgmap v624: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:09.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:08 smithi103 ceph-mon[32635]: pgmap v624: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:09.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:08 smithi204 ceph-mon[32618]: pgmap v624: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:11.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:10 smithi094 ceph-mon[28050]: pgmap v625: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:11.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:10 smithi103 ceph-mon[32635]: pgmap v625: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:11.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:10 smithi204 ceph-mon[32618]: pgmap v625: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:13.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:12 smithi094 ceph-mon[28050]: pgmap v626: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:13.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:12 smithi103 ceph-mon[32635]: pgmap v626: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:13.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:12 smithi204 ceph-mon[32618]: pgmap v626: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:15.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:14 smithi094 ceph-mon[28050]: pgmap v627: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:15.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:14 smithi103 ceph-mon[32635]: pgmap v627: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:15.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:14 smithi204 ceph-mon[32618]: pgmap v627: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:17.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:16 smithi094 ceph-mon[28050]: pgmap v628: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:17.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:16 smithi103 ceph-mon[32635]: pgmap v628: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:17.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:16 smithi204 ceph-mon[32618]: pgmap v628: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:19.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:18 smithi094 ceph-mon[28050]: pgmap v629: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:19.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:18 smithi103 ceph-mon[32635]: pgmap v629: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:19.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:18 smithi204 ceph-mon[32618]: pgmap v629: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:21.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:20 smithi094 ceph-mon[28050]: pgmap v630: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:21.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:20 smithi103 ceph-mon[32635]: pgmap v630: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:21.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:20 smithi204 ceph-mon[32618]: pgmap v630: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:23.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:22 smithi094 ceph-mon[28050]: pgmap v631: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:23.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:22 smithi103 ceph-mon[32635]: pgmap v631: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:23.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:22 smithi204 ceph-mon[32618]: pgmap v631: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:25.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:24 smithi094 ceph-mon[28050]: pgmap v632: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:25.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:24 smithi103 ceph-mon[32635]: pgmap v632: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:25.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:24 smithi204 ceph-mon[32618]: pgmap v632: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:27.184 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:26 smithi204 ceph-mon[32618]: pgmap v633: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:27.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:26 smithi094 ceph-mon[28050]: pgmap v633: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:27.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:26 smithi103 ceph-mon[32635]: pgmap v633: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:29.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:28 smithi094 ceph-mon[28050]: pgmap v634: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:29.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:28 smithi103 ceph-mon[32635]: pgmap v634: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:29.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:28 smithi204 ceph-mon[32618]: pgmap v634: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:31.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:30 smithi094 ceph-mon[28050]: pgmap v635: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:31.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:30 smithi103 ceph-mon[32635]: pgmap v635: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:31.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:30 smithi204 ceph-mon[32618]: pgmap v635: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:33.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:32 smithi094 ceph-mon[28050]: pgmap v636: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:33.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:32 smithi103 ceph-mon[32635]: pgmap v636: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:33.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:32 smithi204 ceph-mon[32618]: pgmap v636: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:35.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:18:35.023+0000 7f3256000640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f3238005850 con 0x7f326010a990 2024-04-26T20:18:35.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:18:35.024+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f3238005850 con 0x7f326010a990 2024-04-26T20:18:35.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:34 smithi094 ceph-mon[28050]: pgmap v637: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:35.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:34 smithi103 ceph-mon[32635]: pgmap v637: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:35.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:34 smithi204 ceph-mon[32618]: pgmap v637: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:37.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:36 smithi094 ceph-mon[28050]: pgmap v638: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:37.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:36 smithi103 ceph-mon[32635]: pgmap v638: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:37.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:36 smithi204 ceph-mon[32618]: pgmap v638: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:39.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:38 smithi094 ceph-mon[28050]: pgmap v639: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:39.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:38 smithi103 ceph-mon[32635]: pgmap v639: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:39.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:38 smithi204 ceph-mon[32618]: pgmap v639: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:41.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:40 smithi094 ceph-mon[28050]: pgmap v640: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:41.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:40 smithi103 ceph-mon[32635]: pgmap v640: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:41.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:40 smithi204 ceph-mon[32618]: pgmap v640: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:42 smithi094 ceph-mon[28050]: pgmap v641: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:43.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:42 smithi103 ceph-mon[32635]: pgmap v641: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:43.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:42 smithi204 ceph-mon[32618]: pgmap v641: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:45.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:44 smithi094 ceph-mon[28050]: pgmap v642: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:45.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:44 smithi103 ceph-mon[32635]: pgmap v642: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:45.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:44 smithi204 ceph-mon[32618]: pgmap v642: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:47.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:46 smithi103 ceph-mon[32635]: pgmap v643: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:47.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:46 smithi204 ceph-mon[32618]: pgmap v643: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:47.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:46 smithi094 ceph-mon[28050]: pgmap v643: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:49.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:48 smithi103 ceph-mon[32635]: pgmap v644: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:49.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:48 smithi204 ceph-mon[32618]: pgmap v644: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:49.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:48 smithi094 ceph-mon[28050]: pgmap v644: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T20:18:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:50 smithi103 ceph-mon[32635]: pgmap v645: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:51.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:50 smithi204 ceph-mon[32618]: pgmap v645: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:51.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:50 smithi094 ceph-mon[28050]: pgmap v645: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:53 smithi103 ceph-mon[32635]: pgmap v646: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:53.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:53 smithi204 ceph-mon[32618]: pgmap v646: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:53.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:53 smithi094 ceph-mon[28050]: pgmap v646: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:18:55.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:55 smithi103 ceph-mon[32635]: pgmap v647: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:55.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:55 smithi204 ceph-mon[32618]: pgmap v647: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:55.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:55 smithi094 ceph-mon[28050]: pgmap v647: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:57.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:57 smithi103 ceph-mon[32635]: pgmap v648: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:57.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:57 smithi204 ceph-mon[32618]: pgmap v648: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:57.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:57 smithi094 ceph-mon[28050]: pgmap v648: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:18:59.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:18:59 smithi103 ceph-mon[32635]: pgmap v649: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:59.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:18:59 smithi204 ceph-mon[32618]: pgmap v649: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:18:59.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:18:59 smithi094 ceph-mon[28050]: pgmap v649: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:02.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:01 smithi103 ceph-mon[32635]: pgmap v650: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:02.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:01 smithi204 ceph-mon[32618]: pgmap v650: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:01 smithi094 ceph-mon[28050]: pgmap v650: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:04.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:03 smithi103 ceph-mon[32635]: pgmap v651: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:04.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:03 smithi204 ceph-mon[32618]: pgmap v651: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:04.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:03 smithi094 ceph-mon[28050]: pgmap v651: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:06.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:05 smithi103 ceph-mon[32635]: pgmap v652: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:06.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:05 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:19:06.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:05 smithi204 ceph-mon[32618]: pgmap v652: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:06.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:05 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:19:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:05 smithi094 ceph-mon[28050]: pgmap v652: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:05 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:19:07.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:19:07.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:19:07.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:19:07.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:19:07.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:19:07.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:19:07.285 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:19:07.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:19:07.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:19:08.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:07 smithi204 ceph-mon[32618]: pgmap v653: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:08.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:07 smithi094 ceph-mon[28050]: pgmap v653: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:08.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:07 smithi103 ceph-mon[32635]: pgmap v653: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:10.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:09 smithi204 ceph-mon[32618]: pgmap v654: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:10.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:09 smithi094 ceph-mon[28050]: pgmap v654: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:10.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:09 smithi103 ceph-mon[32635]: pgmap v654: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:12.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:11 smithi204 ceph-mon[32618]: pgmap v655: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:11 smithi094 ceph-mon[28050]: pgmap v655: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:12.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:11 smithi103 ceph-mon[32635]: pgmap v655: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:14.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:13 smithi204 ceph-mon[32618]: pgmap v656: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:14.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:13 smithi094 ceph-mon[28050]: pgmap v656: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:14.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:13 smithi103 ceph-mon[32635]: pgmap v656: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:16.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:15 smithi204 ceph-mon[32618]: pgmap v657: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:16.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:15 smithi094 ceph-mon[28050]: pgmap v657: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:16.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:15 smithi103 ceph-mon[32635]: pgmap v657: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:18.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:17 smithi204 ceph-mon[32618]: pgmap v658: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:18.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:17 smithi094 ceph-mon[28050]: pgmap v658: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:18.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:17 smithi103 ceph-mon[32635]: pgmap v658: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:20.086 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:19 smithi103 ceph-mon[32635]: pgmap v659: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:20.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:19 smithi204 ceph-mon[32618]: pgmap v659: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:20.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:19 smithi094 ceph-mon[28050]: pgmap v659: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:22.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:21 smithi204 ceph-mon[32618]: pgmap v660: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T20:19:22.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:21 smithi094 ceph-mon[28050]: pgmap v660: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T20:19:22.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:21 smithi103 ceph-mon[32635]: pgmap v660: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T20:19:24.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:23 smithi204 ceph-mon[32618]: pgmap v661: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:19:24.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:23 smithi094 ceph-mon[28050]: pgmap v661: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:19:24.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:23 smithi103 ceph-mon[32635]: pgmap v661: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T20:19:26.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:25 smithi204 ceph-mon[32618]: pgmap v662: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T20:19:26.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:25 smithi094 ceph-mon[28050]: pgmap v662: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T20:19:26.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:25 smithi103 ceph-mon[32635]: pgmap v662: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T20:19:28.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:27 smithi204 ceph-mon[32618]: pgmap v663: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T20:19:28.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:27 smithi094 ceph-mon[28050]: pgmap v663: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T20:19:28.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:27 smithi103 ceph-mon[32635]: pgmap v663: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T20:19:30.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:29 smithi204 ceph-mon[32618]: pgmap v664: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T20:19:30.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:29 smithi094 ceph-mon[28050]: pgmap v664: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T20:19:30.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:29 smithi103 ceph-mon[32635]: pgmap v664: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T20:19:32.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:31 smithi204 ceph-mon[32618]: pgmap v665: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T20:19:32.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:31 smithi094 ceph-mon[28050]: pgmap v665: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T20:19:32.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:31 smithi103 ceph-mon[32635]: pgmap v665: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-26T20:19:33.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:32 smithi094 ceph-mon[28050]: pgmap v666: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:33.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:32 smithi103 ceph-mon[32635]: pgmap v666: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:33.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:32 smithi204 ceph-mon[32618]: pgmap v666: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:35.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:34 smithi094 ceph-mon[28050]: pgmap v667: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:35.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:34 smithi103 ceph-mon[32635]: pgmap v667: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:35.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:34 smithi204 ceph-mon[32618]: pgmap v667: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:37.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:36 smithi094 ceph-mon[28050]: pgmap v668: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:37.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:36 smithi103 ceph-mon[32635]: pgmap v668: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:37.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:36 smithi204 ceph-mon[32618]: pgmap v668: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:39.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:38 smithi094 ceph-mon[28050]: pgmap v669: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:39.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:38 smithi103 ceph-mon[32635]: pgmap v669: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:39.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:38 smithi204 ceph-mon[32618]: pgmap v669: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:41.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:40 smithi094 ceph-mon[28050]: pgmap v670: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:41.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:40 smithi103 ceph-mon[32635]: pgmap v670: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:41.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:40 smithi204 ceph-mon[32618]: pgmap v670: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:43.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:42 smithi094 ceph-mon[28050]: pgmap v671: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:43.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:42 smithi103 ceph-mon[32635]: pgmap v671: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:43.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:42 smithi204 ceph-mon[32618]: pgmap v671: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:45.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:44 smithi094 ceph-mon[28050]: pgmap v672: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:45.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:44 smithi103 ceph-mon[32635]: pgmap v672: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:45.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:44 smithi204 ceph-mon[32618]: pgmap v672: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:47.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:46 smithi094 ceph-mon[28050]: pgmap v673: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:47.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:46 smithi103 ceph-mon[32635]: pgmap v673: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:47.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:46 smithi204 ceph-mon[32618]: pgmap v673: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:49.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:48 smithi094 ceph-mon[28050]: pgmap v674: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:49.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:48 smithi103 ceph-mon[32635]: pgmap v674: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:49.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:48 smithi204 ceph-mon[32618]: pgmap v674: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:51.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:50 smithi094 ceph-mon[28050]: pgmap v675: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:51.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:50 smithi103 ceph-mon[32635]: pgmap v675: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:51.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:50 smithi204 ceph-mon[32618]: pgmap v675: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:53.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:52 smithi094 ceph-mon[28050]: pgmap v676: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:53.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:52 smithi103 ceph-mon[32635]: pgmap v676: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:53.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:52 smithi204 ceph-mon[32618]: pgmap v676: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:19:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:54 smithi094 ceph-mon[28050]: pgmap v677: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:55.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:54 smithi103 ceph-mon[32635]: pgmap v677: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:55.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:54 smithi204 ceph-mon[32618]: pgmap v677: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:57.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:56 smithi094 ceph-mon[28050]: pgmap v678: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:57.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:56 smithi103 ceph-mon[32635]: pgmap v678: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:57.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:56 smithi204 ceph-mon[32618]: pgmap v678: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:19:59.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:19:58 smithi094 ceph-mon[28050]: pgmap v679: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:59.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:19:58 smithi103 ceph-mon[32635]: pgmap v679: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:19:59.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:19:58 smithi204 ceph-mon[32618]: pgmap v679: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:01.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:00 smithi094 ceph-mon[28050]: pgmap v680: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:01.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:00 smithi094 ceph-mon[28050]: overall HEALTH_OK 2024-04-26T20:20:01.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:00 smithi103 ceph-mon[32635]: pgmap v680: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:01.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:00 smithi103 ceph-mon[32635]: overall HEALTH_OK 2024-04-26T20:20:01.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:00 smithi204 ceph-mon[32618]: pgmap v680: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:01.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:00 smithi204 ceph-mon[32618]: overall HEALTH_OK 2024-04-26T20:20:03.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:02 smithi094 ceph-mon[28050]: pgmap v681: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:03.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:02 smithi103 ceph-mon[32635]: pgmap v681: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:03.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:02 smithi204 ceph-mon[32618]: pgmap v681: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:05.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:04 smithi094 ceph-mon[28050]: pgmap v682: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:05.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:04 smithi103 ceph-mon[32635]: pgmap v682: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:05.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:04 smithi204 ceph-mon[32618]: pgmap v682: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:07.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:06 smithi103 ceph-mon[32635]: pgmap v683: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:07.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:06 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:20:07.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:06 smithi204 ceph-mon[32618]: pgmap v683: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:07.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:06 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:20:07.486 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:06 smithi094 ceph-mon[28050]: pgmap v683: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:07.486 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:06 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:20:09.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:08 smithi103 ceph-mon[32635]: pgmap v684: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:09.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:20:09.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:20:09.286 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:20:09.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:08 smithi204 ceph-mon[32618]: pgmap v684: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:09.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:20:09.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:20:09.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:20:09.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:08 smithi094 ceph-mon[28050]: pgmap v684: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:09.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:20:09.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:20:09.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:20:11.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:10 smithi103 ceph-mon[32635]: pgmap v685: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:11.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:10 smithi204 ceph-mon[32618]: pgmap v685: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:11.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:10 smithi094 ceph-mon[28050]: pgmap v685: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:13.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:13 smithi103 ceph-mon[32635]: pgmap v686: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:13.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:13 smithi204 ceph-mon[32618]: pgmap v686: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:13.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:13 smithi094 ceph-mon[28050]: pgmap v686: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:15.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:15 smithi103 ceph-mon[32635]: pgmap v687: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:15.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:15 smithi204 ceph-mon[32618]: pgmap v687: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:15.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:15 smithi094 ceph-mon[28050]: pgmap v687: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:17.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:17 smithi103 ceph-mon[32635]: pgmap v688: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:17.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:17 smithi204 ceph-mon[32618]: pgmap v688: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:17.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:17 smithi094 ceph-mon[28050]: pgmap v688: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:19.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:19 smithi103 ceph-mon[32635]: pgmap v689: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:19.290 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:19 smithi094 ceph-mon[28050]: pgmap v689: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:19.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:19 smithi204 ceph-mon[32618]: pgmap v689: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:21.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:21 smithi103 ceph-mon[32635]: pgmap v690: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:21.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:21 smithi204 ceph-mon[32618]: pgmap v690: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:21.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:21 smithi094 ceph-mon[28050]: pgmap v690: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:23.281 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:23 smithi103 ceph-mon[32635]: pgmap v691: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:23.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:23 smithi204 ceph-mon[32618]: pgmap v691: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:23 smithi094 ceph-mon[28050]: pgmap v691: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:25.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:25 smithi204 ceph-mon[32618]: pgmap v692: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:25.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:25 smithi094 ceph-mon[28050]: pgmap v692: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:25.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:25 smithi103 ceph-mon[32635]: pgmap v692: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:27.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:27 smithi204 ceph-mon[32618]: pgmap v693: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:27.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:27 smithi094 ceph-mon[28050]: pgmap v693: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:27.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:27 smithi103 ceph-mon[32635]: pgmap v693: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:29.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:29 smithi204 ceph-mon[32618]: pgmap v694: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:29.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:29 smithi094 ceph-mon[28050]: pgmap v694: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:29.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:29 smithi103 ceph-mon[32635]: pgmap v694: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:31.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:31 smithi204 ceph-mon[32618]: pgmap v695: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:31.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:31 smithi094 ceph-mon[28050]: pgmap v695: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:31.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:31 smithi103 ceph-mon[32635]: pgmap v695: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:33.350 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:33 smithi103 ceph-mon[32635]: pgmap v696: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:33.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:33 smithi204 ceph-mon[32618]: pgmap v696: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:33.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:33 smithi094 ceph-mon[28050]: pgmap v696: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:35.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:35 smithi204 ceph-mon[32618]: pgmap v697: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:35.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:35 smithi094 ceph-mon[28050]: pgmap v697: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:35.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:35 smithi103 ceph-mon[32635]: pgmap v697: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:37.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:37 smithi204 ceph-mon[32618]: pgmap v698: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:37.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:37 smithi094 ceph-mon[28050]: pgmap v698: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:37 smithi103 ceph-mon[32635]: pgmap v698: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:39.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:39 smithi204 ceph-mon[32618]: pgmap v699: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:39.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:39 smithi094 ceph-mon[28050]: pgmap v699: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:39.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:39 smithi103 ceph-mon[32635]: pgmap v699: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:41.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:41 smithi204 ceph-mon[32618]: pgmap v700: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:41.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:41 smithi094 ceph-mon[28050]: pgmap v700: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:41.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:41 smithi103 ceph-mon[32635]: pgmap v700: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:43.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:43 smithi204 ceph-mon[32618]: pgmap v701: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:43.385 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:43 smithi094 ceph-mon[28050]: pgmap v701: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:43.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:43 smithi103 ceph-mon[32635]: pgmap v701: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:45.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:45 smithi204 ceph-mon[32618]: pgmap v702: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:45.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:45 smithi094 ceph-mon[28050]: pgmap v702: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:45.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:45 smithi103 ceph-mon[32635]: pgmap v702: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:47.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:47 smithi204 ceph-mon[32618]: pgmap v703: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:47.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:47 smithi094 ceph-mon[28050]: pgmap v703: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:47.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:47 smithi103 ceph-mon[32635]: pgmap v703: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:49.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:49 smithi204 ceph-mon[32618]: pgmap v704: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:49.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:49 smithi094 ceph-mon[28050]: pgmap v704: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:49.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:49 smithi103 ceph-mon[32635]: pgmap v704: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:50.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:20:50.045+0000 7f3256000640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f3238005b70 con 0x7f326010a990 2024-04-26T20:20:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:20:50.046+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f3238005b70 con 0x7f326010a990 2024-04-26T20:20:51.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:51 smithi204 ceph-mon[32618]: pgmap v705: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:51.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:51 smithi094 ceph-mon[28050]: pgmap v705: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:51.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:51 smithi103 ceph-mon[32635]: pgmap v705: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:53.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:53 smithi204 ceph-mon[32618]: pgmap v706: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:53.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:53 smithi094 ceph-mon[28050]: pgmap v706: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:53.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:53 smithi103 ceph-mon[32635]: pgmap v706: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:20:55.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:55 smithi204 ceph-mon[32618]: pgmap v707: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:55.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:55 smithi094 ceph-mon[28050]: pgmap v707: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:55.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:55 smithi103 ceph-mon[32635]: pgmap v707: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:57.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:57 smithi204 ceph-mon[32618]: pgmap v708: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:57.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:57 smithi094 ceph-mon[28050]: pgmap v708: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:57.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:57 smithi103 ceph-mon[32635]: pgmap v708: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:20:59.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:20:59 smithi204 ceph-mon[32618]: pgmap v709: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:59.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:20:59 smithi094 ceph-mon[28050]: pgmap v709: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:20:59.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:20:59 smithi103 ceph-mon[32635]: pgmap v709: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:02.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:01 smithi204 ceph-mon[32618]: pgmap v710: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:01 smithi094 ceph-mon[28050]: pgmap v710: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:01 smithi103 ceph-mon[32635]: pgmap v710: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:04.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:03 smithi204 ceph-mon[32618]: pgmap v711: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:04.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:03 smithi094 ceph-mon[28050]: pgmap v711: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:03 smithi103 ceph-mon[32635]: pgmap v711: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:06.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:05 smithi204 ceph-mon[32618]: pgmap v712: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:05 smithi094 ceph-mon[28050]: pgmap v712: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:06.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:05 smithi103 ceph-mon[32635]: pgmap v712: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:08.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:07 smithi204 ceph-mon[32618]: pgmap v713: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:08.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:07 smithi094 ceph-mon[28050]: pgmap v713: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:08.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:07 smithi103 ceph-mon[32635]: pgmap v713: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:09.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:08 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:21:09.190 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:08 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:21:09.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:08 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:21:10.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:09 smithi204 ceph-mon[32618]: pgmap v714: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:10.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:10.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:09 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:10.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:09 smithi094 ceph-mon[28050]: pgmap v714: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:10.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:10.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:09 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:10.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:09 smithi103 ceph-mon[32635]: pgmap v714: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:10.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:10.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:09 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:11 smithi204 ceph-mon[32618]: pgmap v715: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:11.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:11 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:11 smithi094 ceph-mon[28050]: pgmap v715: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:11.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:11 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:11 smithi103 ceph-mon[32635]: pgmap v715: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:11.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:11.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:11 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:13.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:13 smithi204 ceph-mon[32618]: pgmap v716: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:13.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:13.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:13 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:13.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:13 smithi094 ceph-mon[28050]: pgmap v716: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:13.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:13.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:13 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:13.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:13 smithi103 ceph-mon[32635]: pgmap v716: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:13.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:13.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:13 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:14.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:21:14.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:21:14.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:14.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:21:14.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:21:14.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:14.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:21:14.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:21:14.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:21:15.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:15 smithi204 ceph-mon[32618]: pgmap v717: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:15.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:15 smithi094 ceph-mon[28050]: pgmap v717: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:15.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:15 smithi103 ceph-mon[32635]: pgmap v717: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:17.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:17 smithi204 ceph-mon[32618]: pgmap v718: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:17.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:17 smithi094 ceph-mon[28050]: pgmap v718: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:17.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:17 smithi103 ceph-mon[32635]: pgmap v718: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:19.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:19 smithi204 ceph-mon[32618]: pgmap v719: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:19.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:19 smithi094 ceph-mon[28050]: pgmap v719: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:19.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:19 smithi103 ceph-mon[32635]: pgmap v719: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:21.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:21 smithi094 ceph-mon[28050]: pgmap v720: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:21.586 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:21 smithi103 ceph-mon[32635]: pgmap v720: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:21.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:21 smithi204 ceph-mon[32618]: pgmap v720: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:23.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:23 smithi204 ceph-mon[32618]: pgmap v721: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:23.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:23 smithi094 ceph-mon[28050]: pgmap v721: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:23.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:23 smithi103 ceph-mon[32635]: pgmap v721: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:25.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:25 smithi204 ceph-mon[32618]: pgmap v722: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:25.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:25 smithi094 ceph-mon[28050]: pgmap v722: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:25.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:25 smithi103 ceph-mon[32635]: pgmap v722: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:27.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:27 smithi204 ceph-mon[32618]: pgmap v723: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:27 smithi094 ceph-mon[28050]: pgmap v723: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:27.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:27 smithi103 ceph-mon[32635]: pgmap v723: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:29.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:29 smithi204 ceph-mon[32618]: pgmap v724: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:29 smithi094 ceph-mon[28050]: pgmap v724: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:29.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:29 smithi103 ceph-mon[32635]: pgmap v724: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:31.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:31 smithi094 ceph-mon[28050]: pgmap v725: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:31.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:31 smithi204 ceph-mon[32618]: pgmap v725: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:31.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:31 smithi103 ceph-mon[32635]: pgmap v725: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:33.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:33 smithi204 ceph-mon[32618]: pgmap v726: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:33.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:33 smithi094 ceph-mon[28050]: pgmap v726: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:33.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:33 smithi103 ceph-mon[32635]: pgmap v726: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:35.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:21:35.052+0000 7f3256000640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f3238005ed0 con 0x7f326010a990 2024-04-26T20:21:35.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:21:35.053+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f3238005ed0 con 0x7f326010a990 2024-04-26T20:21:35.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:35 smithi204 ceph-mon[32618]: pgmap v727: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:35.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:35 smithi094 ceph-mon[28050]: pgmap v727: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:35.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:35 smithi103 ceph-mon[32635]: pgmap v727: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:37.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:37 smithi204 ceph-mon[32618]: pgmap v728: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:37.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:37 smithi094 ceph-mon[28050]: pgmap v728: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:37.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:37 smithi103 ceph-mon[32635]: pgmap v728: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:39.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:39 smithi204 ceph-mon[32618]: pgmap v729: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:39.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:39 smithi094 ceph-mon[28050]: pgmap v729: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:39.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:39 smithi103 ceph-mon[32635]: pgmap v729: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:41.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:41 smithi204 ceph-mon[32618]: pgmap v730: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:41.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:41 smithi094 ceph-mon[28050]: pgmap v730: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:41 smithi103 ceph-mon[32635]: pgmap v730: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:43.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:43 smithi094 ceph-mon[28050]: pgmap v731: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:43.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:43 smithi103 ceph-mon[32635]: pgmap v731: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:43.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:43 smithi204 ceph-mon[32618]: pgmap v731: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:21:45.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:45 smithi094 ceph-mon[28050]: pgmap v732: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:45.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:45 smithi103 ceph-mon[32635]: pgmap v732: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:45.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:45 smithi204 ceph-mon[32618]: pgmap v732: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:21:47.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:47 smithi094 ceph-mon[28050]: pgmap v733: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:47.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:47 smithi103 ceph-mon[32635]: pgmap v733: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:47.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:47 smithi204 ceph-mon[32618]: pgmap v733: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:21:49.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:49 smithi094 ceph-mon[28050]: pgmap v734: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:21:49.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:49 smithi103 ceph-mon[32635]: pgmap v734: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:21:49.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:49 smithi204 ceph-mon[32618]: pgmap v734: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:21:51.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:51 smithi094 ceph-mon[28050]: pgmap v735: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:21:51.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:51 smithi103 ceph-mon[32635]: pgmap v735: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:21:51.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:51 smithi204 ceph-mon[32618]: pgmap v735: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:21:53.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:53 smithi094 ceph-mon[28050]: pgmap v736: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:21:53.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:53 smithi103 ceph-mon[32635]: pgmap v736: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:21:53.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:53 smithi204 ceph-mon[32618]: pgmap v736: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T20:21:55.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:55 smithi094 ceph-mon[28050]: pgmap v737: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:21:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:55 smithi103 ceph-mon[32635]: pgmap v737: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:21:55.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:55 smithi204 ceph-mon[32618]: pgmap v737: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:21:57.684 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:57 smithi204 ceph-mon[32618]: pgmap v738: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:21:57.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:57 smithi094 ceph-mon[28050]: pgmap v738: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:21:57.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:57 smithi103 ceph-mon[32635]: pgmap v738: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T20:21:59.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:21:59 smithi094 ceph-mon[28050]: pgmap v739: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:21:59.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:21:59 smithi103 ceph-mon[32635]: pgmap v739: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:21:59.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:21:59 smithi204 ceph-mon[32618]: pgmap v739: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-26T20:22:01.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:01 smithi094 ceph-mon[28050]: pgmap v740: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:01.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:01 smithi103 ceph-mon[32635]: pgmap v740: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:01.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:01 smithi204 ceph-mon[32618]: pgmap v740: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:03.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:03 smithi094 ceph-mon[28050]: pgmap v741: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:03.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:03 smithi103 ceph-mon[32635]: pgmap v741: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:03.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:03 smithi204 ceph-mon[32618]: pgmap v741: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:05.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:05 smithi094 ceph-mon[28050]: pgmap v742: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:05.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:05 smithi103 ceph-mon[32635]: pgmap v742: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:05.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:05 smithi204 ceph-mon[32618]: pgmap v742: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:07.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:07 smithi094 ceph-mon[28050]: pgmap v743: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:07.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:07 smithi103 ceph-mon[32635]: pgmap v743: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:07.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:07 smithi204 ceph-mon[32618]: pgmap v743: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:09.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:09 smithi094 ceph-mon[28050]: pgmap v744: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:09.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:09 smithi103 ceph-mon[32635]: pgmap v744: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:09.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:09 smithi204 ceph-mon[32618]: pgmap v744: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:11 smithi094 ceph-mon[28050]: pgmap v745: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:11.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:11 smithi103 ceph-mon[32635]: pgmap v745: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:11.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:11 smithi204 ceph-mon[32618]: pgmap v745: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:13.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:13 smithi094 ceph-mon[28050]: pgmap v746: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:13.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:13 smithi103 ceph-mon[32635]: pgmap v746: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:13.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:13 smithi204 ceph-mon[32618]: pgmap v746: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:14.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:22:14.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:14 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:22:14.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:14 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:22:14.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:14.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:14 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:15.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:15 smithi094 ceph-mon[28050]: pgmap v747: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:15.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:22:15.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:22:15.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:15.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:15 smithi103 ceph-mon[32635]: pgmap v747: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:15.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:22:15.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:22:15.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:15.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:15 smithi204 ceph-mon[32618]: pgmap v747: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:15.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:22:15.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:22:15.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:22:17.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:17 smithi094 ceph-mon[28050]: pgmap v748: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:17.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:17 smithi103 ceph-mon[32635]: pgmap v748: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:17.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:17 smithi204 ceph-mon[32618]: pgmap v748: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:19.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:19 smithi094 ceph-mon[28050]: pgmap v749: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:19.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:19 smithi103 ceph-mon[32635]: pgmap v749: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:19.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:19 smithi204 ceph-mon[32618]: pgmap v749: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:21.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:21 smithi103 ceph-mon[32635]: pgmap v750: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:21.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:21 smithi204 ceph-mon[32618]: pgmap v750: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:21.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:21 smithi094 ceph-mon[28050]: pgmap v750: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:23.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:23 smithi103 ceph-mon[32635]: pgmap v751: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:23.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:23 smithi204 ceph-mon[32618]: pgmap v751: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:23 smithi094 ceph-mon[28050]: pgmap v751: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:25.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:25 smithi103 ceph-mon[32635]: pgmap v752: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:25.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:25 smithi204 ceph-mon[32618]: pgmap v752: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:25 smithi094 ceph-mon[28050]: pgmap v752: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:27.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:27 smithi103 ceph-mon[32635]: pgmap v753: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:27.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:27 smithi204 ceph-mon[32618]: pgmap v753: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:27.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:27 smithi094 ceph-mon[28050]: pgmap v753: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:29.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:29 smithi103 ceph-mon[32635]: pgmap v754: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:29.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:29 smithi204 ceph-mon[32618]: pgmap v754: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:29.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:29 smithi094 ceph-mon[28050]: pgmap v754: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:31.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:31 smithi103 ceph-mon[32635]: pgmap v755: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:31.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:31 smithi204 ceph-mon[32618]: pgmap v755: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:31.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:31 smithi094 ceph-mon[28050]: pgmap v755: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:33.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:33 smithi103 ceph-mon[32635]: pgmap v756: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:33.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:33 smithi204 ceph-mon[32618]: pgmap v756: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:33.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:33 smithi094 ceph-mon[28050]: pgmap v756: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:35.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:35 smithi204 ceph-mon[32618]: pgmap v757: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:35.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:35 smithi094 ceph-mon[28050]: pgmap v757: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:36.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:35 smithi103 ceph-mon[32635]: pgmap v757: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:37.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:37 smithi204 ceph-mon[32618]: pgmap v758: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:37.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:37 smithi094 ceph-mon[28050]: pgmap v758: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:37 smithi103 ceph-mon[32635]: pgmap v758: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:39.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:39 smithi204 ceph-mon[32618]: pgmap v759: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:39.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:39 smithi094 ceph-mon[28050]: pgmap v759: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:40.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:39 smithi103 ceph-mon[32635]: pgmap v759: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:41.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:41 smithi204 ceph-mon[32618]: pgmap v760: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:41.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:41 smithi094 ceph-mon[28050]: pgmap v760: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:42.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:41 smithi103 ceph-mon[32635]: pgmap v760: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:43.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:43 smithi204 ceph-mon[32618]: pgmap v761: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:43.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:43 smithi094 ceph-mon[28050]: pgmap v761: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:44.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:43 smithi103 ceph-mon[32635]: pgmap v761: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:45.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:45 smithi204 ceph-mon[32618]: pgmap v762: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:45.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:45 smithi094 ceph-mon[28050]: pgmap v762: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:46.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:45 smithi103 ceph-mon[32635]: pgmap v762: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:47.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:47 smithi204 ceph-mon[32618]: pgmap v763: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:47.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:47 smithi094 ceph-mon[28050]: pgmap v763: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:48.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:47 smithi103 ceph-mon[32635]: pgmap v763: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:49.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:49 smithi204 ceph-mon[32618]: pgmap v764: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:49.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:49 smithi094 ceph-mon[28050]: pgmap v764: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:50.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:49 smithi103 ceph-mon[32635]: pgmap v764: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:51.836 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:51 smithi103 ceph-mon[32635]: pgmap v765: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:51.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:51 smithi204 ceph-mon[32618]: pgmap v765: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:51.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:51 smithi094 ceph-mon[28050]: pgmap v765: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:53.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:53 smithi204 ceph-mon[32618]: pgmap v766: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:53.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:53 smithi094 ceph-mon[28050]: pgmap v766: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:54.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:53 smithi103 ceph-mon[32635]: pgmap v766: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:22:55.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:55 smithi204 ceph-mon[32618]: pgmap v767: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:55.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:55 smithi094 ceph-mon[28050]: pgmap v767: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:56.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:55 smithi103 ceph-mon[32635]: pgmap v767: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:57.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:57 smithi204 ceph-mon[32618]: pgmap v768: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:57.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:57 smithi094 ceph-mon[28050]: pgmap v768: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:58.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:57 smithi103 ceph-mon[32635]: pgmap v768: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:22:59.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:22:59 smithi204 ceph-mon[32618]: pgmap v769: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:22:59.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:22:59 smithi094 ceph-mon[28050]: pgmap v769: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:00.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:22:59 smithi103 ceph-mon[32635]: pgmap v769: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:02.132 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:01 smithi204 ceph-mon[32618]: pgmap v770: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:01 smithi094 ceph-mon[28050]: pgmap v770: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:01 smithi103 ceph-mon[32635]: pgmap v770: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:04.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:03 smithi204 ceph-mon[32618]: pgmap v771: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:04.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:03 smithi094 ceph-mon[28050]: pgmap v771: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:03 smithi103 ceph-mon[32635]: pgmap v771: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:06.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:05 smithi204 ceph-mon[32618]: pgmap v772: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:06.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:05 smithi094 ceph-mon[28050]: pgmap v772: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:06.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:05 smithi103 ceph-mon[32635]: pgmap v772: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:08.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:07 smithi204 ceph-mon[32618]: pgmap v773: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:08.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:07 smithi094 ceph-mon[28050]: pgmap v773: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:08.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:07 smithi103 ceph-mon[32635]: pgmap v773: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:10.134 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:09 smithi204 ceph-mon[32618]: pgmap v774: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:10.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:09 smithi094 ceph-mon[28050]: pgmap v774: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:10.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:09 smithi103 ceph-mon[32635]: pgmap v774: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:12.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:11 smithi204 ceph-mon[32618]: pgmap v775: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:11 smithi094 ceph-mon[28050]: pgmap v775: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:12.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:11 smithi103 ceph-mon[32635]: pgmap v775: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:14.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:13 smithi204 ceph-mon[32618]: pgmap v776: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:14.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:13 smithi094 ceph-mon[28050]: pgmap v776: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:14.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:13 smithi103 ceph-mon[32635]: pgmap v776: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:16.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:15 smithi094 ceph-mon[28050]: pgmap v777: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:16.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:15 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:16.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:15 smithi103 ceph-mon[32635]: pgmap v777: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:16.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:15 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:16.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:15 smithi204 ceph-mon[32618]: pgmap v777: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:16.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:15 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:17.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:16 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:23:17.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:16 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:23:17.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:16 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:23:17.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:16 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:23:17.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:16 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:23:17.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:16 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:23:17.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:16 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:23:17.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:16 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:23:17.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:16 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:23:18.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:17 smithi094 ceph-mon[28050]: pgmap v778: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:18.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:17 smithi103 ceph-mon[32635]: pgmap v778: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:18.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:17 smithi204 ceph-mon[32618]: pgmap v778: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:19.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:18 smithi094 ceph-mon[28050]: pgmap v779: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:19.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:18 smithi103 ceph-mon[32635]: pgmap v779: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:19.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:18 smithi204 ceph-mon[32618]: pgmap v779: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:21.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:21 smithi204 ceph-mon[32618]: pgmap v780: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:21.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:21 smithi094 ceph-mon[28050]: pgmap v780: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:21.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:21 smithi103 ceph-mon[32635]: pgmap v780: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:23.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:23 smithi204 ceph-mon[32618]: pgmap v781: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:23.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:23 smithi094 ceph-mon[28050]: pgmap v781: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:23.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:23 smithi103 ceph-mon[32635]: pgmap v781: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:25.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:25 smithi204 ceph-mon[32618]: pgmap v782: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:25.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:25 smithi094 ceph-mon[28050]: pgmap v782: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:25.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:25 smithi103 ceph-mon[32635]: pgmap v782: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:27.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:27 smithi204 ceph-mon[32618]: pgmap v783: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:27.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:27 smithi094 ceph-mon[28050]: pgmap v783: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:27.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:27 smithi103 ceph-mon[32635]: pgmap v783: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:29.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:29 smithi204 ceph-mon[32618]: pgmap v784: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:29.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:29 smithi094 ceph-mon[28050]: pgmap v784: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:29.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:29 smithi103 ceph-mon[32635]: pgmap v784: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:31.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:31 smithi204 ceph-mon[32618]: pgmap v785: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:31.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:31 smithi094 ceph-mon[28050]: pgmap v785: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:31.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:31 smithi103 ceph-mon[32635]: pgmap v785: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:33.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:33 smithi204 ceph-mon[32618]: pgmap v786: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:33.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:33 smithi094 ceph-mon[28050]: pgmap v786: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:33.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:33 smithi103 ceph-mon[32635]: pgmap v786: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:35.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:35 smithi204 ceph-mon[32618]: pgmap v787: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:35.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:35 smithi094 ceph-mon[28050]: pgmap v787: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:35.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:35 smithi103 ceph-mon[32635]: pgmap v787: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:37.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:37 smithi204 ceph-mon[32618]: pgmap v788: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:37.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:37 smithi094 ceph-mon[28050]: pgmap v788: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:37 smithi103 ceph-mon[32635]: pgmap v788: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:39.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:39 smithi204 ceph-mon[32618]: pgmap v789: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:39.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:39 smithi094 ceph-mon[28050]: pgmap v789: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:39.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:39 smithi103 ceph-mon[32635]: pgmap v789: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:41.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:41 smithi204 ceph-mon[32618]: pgmap v790: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:41.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:41 smithi094 ceph-mon[28050]: pgmap v790: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:41.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:41 smithi103 ceph-mon[32635]: pgmap v790: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:43.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:43 smithi204 ceph-mon[32618]: pgmap v791: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:43.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:43 smithi094 ceph-mon[28050]: pgmap v791: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:43.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:43 smithi103 ceph-mon[32635]: pgmap v791: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:45.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:45 smithi094 ceph-mon[28050]: pgmap v792: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:45.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:45 smithi103 ceph-mon[32635]: pgmap v792: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:45.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:45 smithi204 ceph-mon[32618]: pgmap v792: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:47.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:47 smithi094 ceph-mon[28050]: pgmap v793: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:47.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:47 smithi103 ceph-mon[32635]: pgmap v793: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:47.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:47 smithi204 ceph-mon[32618]: pgmap v793: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:49.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:49 smithi094 ceph-mon[28050]: pgmap v794: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:49.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:49 smithi103 ceph-mon[32635]: pgmap v794: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:49.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:49 smithi204 ceph-mon[32618]: pgmap v794: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:51.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:51 smithi094 ceph-mon[28050]: pgmap v795: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:51.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:51 smithi103 ceph-mon[32635]: pgmap v795: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:51.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:51 smithi204 ceph-mon[32618]: pgmap v795: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:53.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:53 smithi094 ceph-mon[28050]: pgmap v796: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:53.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:53 smithi103 ceph-mon[32635]: pgmap v796: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:53.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:53 smithi204 ceph-mon[32618]: pgmap v796: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:23:55.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:55 smithi094 ceph-mon[28050]: pgmap v797: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:55.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:55 smithi103 ceph-mon[32635]: pgmap v797: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:55.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:55 smithi204 ceph-mon[32618]: pgmap v797: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:57.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:57 smithi204 ceph-mon[32618]: pgmap v798: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:57.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:57 smithi094 ceph-mon[28050]: pgmap v798: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:57.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:57 smithi103 ceph-mon[32635]: pgmap v798: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:23:59.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:23:59 smithi094 ceph-mon[28050]: pgmap v799: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:59.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:23:59 smithi103 ceph-mon[32635]: pgmap v799: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:23:59.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:23:59 smithi204 ceph-mon[32618]: pgmap v799: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:01.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:01 smithi094 ceph-mon[28050]: pgmap v800: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:01.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:01 smithi103 ceph-mon[32635]: pgmap v800: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:01.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:01 smithi204 ceph-mon[32618]: pgmap v800: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:03.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:03 smithi094 ceph-mon[28050]: pgmap v801: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:03.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:03 smithi103 ceph-mon[32635]: pgmap v801: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:03.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:03 smithi204 ceph-mon[32618]: pgmap v801: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:05.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:05 smithi094 ceph-mon[28050]: pgmap v802: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:05.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:05 smithi103 ceph-mon[32635]: pgmap v802: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:05.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:05 smithi204 ceph-mon[32618]: pgmap v802: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:07.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:07 smithi094 ceph-mon[28050]: pgmap v803: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:07.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:07 smithi103 ceph-mon[32635]: pgmap v803: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:07.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:07 smithi204 ceph-mon[32618]: pgmap v803: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:09.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:09 smithi094 ceph-mon[28050]: pgmap v804: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:09.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:09 smithi103 ceph-mon[32635]: pgmap v804: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:09.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:09 smithi204 ceph-mon[32618]: pgmap v804: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:11.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:11 smithi094 ceph-mon[28050]: pgmap v805: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:11.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:11 smithi103 ceph-mon[32635]: pgmap v805: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:11.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:11 smithi204 ceph-mon[32618]: pgmap v805: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:13.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:13 smithi094 ceph-mon[28050]: pgmap v806: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:13.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:13 smithi103 ceph-mon[32635]: pgmap v806: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:13.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:13 smithi204 ceph-mon[32618]: pgmap v806: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:15.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:15 smithi094 ceph-mon[28050]: pgmap v807: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:15.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:15 smithi103 ceph-mon[32635]: pgmap v807: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:15.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:15 smithi204 ceph-mon[32618]: pgmap v807: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:17.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:17 smithi094 ceph-mon[28050]: pgmap v808: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:17.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:17 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:17.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:17 smithi103 ceph-mon[32635]: pgmap v808: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:17.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:17 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:17.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:17 smithi204 ceph-mon[32618]: pgmap v808: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:17.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:17 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:18.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:18 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:24:18.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:18 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:24:18.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:18 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:24:18.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:18 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:24:18.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:18 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:24:18.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:18 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:24:18.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:18 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:24:18.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:18 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:24:18.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:18 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:24:19.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:19 smithi094 ceph-mon[28050]: pgmap v809: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:19.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:19 smithi103 ceph-mon[32635]: pgmap v809: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:19.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:19 smithi204 ceph-mon[32618]: pgmap v809: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:21.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:21 smithi094 ceph-mon[28050]: pgmap v810: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:21.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:21 smithi103 ceph-mon[32635]: pgmap v810: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:21.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:21 smithi204 ceph-mon[32618]: pgmap v810: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:23.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:23 smithi103 ceph-mon[32635]: pgmap v811: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:23.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:23 smithi204 ceph-mon[32618]: pgmap v811: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:23.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:23 smithi094 ceph-mon[28050]: pgmap v811: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:25.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:25 smithi103 ceph-mon[32635]: pgmap v812: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:25.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:25 smithi204 ceph-mon[32618]: pgmap v812: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:25.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:25 smithi094 ceph-mon[28050]: pgmap v812: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:27.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:27 smithi103 ceph-mon[32635]: pgmap v813: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:27.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:27 smithi204 ceph-mon[32618]: pgmap v813: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:27 smithi094 ceph-mon[28050]: pgmap v813: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:29.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:29 smithi103 ceph-mon[32635]: pgmap v814: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:29.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:29 smithi204 ceph-mon[32618]: pgmap v814: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:29 smithi094 ceph-mon[28050]: pgmap v814: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:31.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:31 smithi103 ceph-mon[32635]: pgmap v815: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:31.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:31 smithi204 ceph-mon[32618]: pgmap v815: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:31 smithi094 ceph-mon[28050]: pgmap v815: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:33.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:33 smithi103 ceph-mon[32635]: pgmap v816: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:33.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:33 smithi204 ceph-mon[32618]: pgmap v816: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:33.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:33 smithi094 ceph-mon[28050]: pgmap v816: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:35.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:24:35.082+0000 7f3256000640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f32380060f0 con 0x7f326010a990 2024-04-26T20:24:35.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:24:35.083+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f32380060f0 con 0x7f326010a990 2024-04-26T20:24:35.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:35 smithi103 ceph-mon[32635]: pgmap v817: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:35.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:35 smithi204 ceph-mon[32618]: pgmap v817: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:35.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:35 smithi094 ceph-mon[28050]: pgmap v817: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:37 smithi103 ceph-mon[32635]: pgmap v818: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:37.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:37 smithi204 ceph-mon[32618]: pgmap v818: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:37.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:37 smithi094 ceph-mon[28050]: pgmap v818: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:39.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:39 smithi103 ceph-mon[32635]: pgmap v819: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:39.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:39 smithi204 ceph-mon[32618]: pgmap v819: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:39.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:39 smithi094 ceph-mon[28050]: pgmap v819: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:41.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:41 smithi103 ceph-mon[32635]: pgmap v820: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:41.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:41 smithi204 ceph-mon[32618]: pgmap v820: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:41.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:41 smithi094 ceph-mon[28050]: pgmap v820: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:43.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:43 smithi204 ceph-mon[32618]: pgmap v821: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:43.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:43 smithi094 ceph-mon[28050]: pgmap v821: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:43.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:43 smithi103 ceph-mon[32635]: pgmap v821: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:45.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:45 smithi204 ceph-mon[32618]: pgmap v822: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:45.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:45 smithi094 ceph-mon[28050]: pgmap v822: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:45.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:45 smithi103 ceph-mon[32635]: pgmap v822: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:47.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:47 smithi204 ceph-mon[32618]: pgmap v823: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:47.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:47 smithi094 ceph-mon[28050]: pgmap v823: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:47.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:47 smithi103 ceph-mon[32635]: pgmap v823: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:49.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:49 smithi204 ceph-mon[32618]: pgmap v824: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:49.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:49 smithi094 ceph-mon[28050]: pgmap v824: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:49.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:49 smithi103 ceph-mon[32635]: pgmap v824: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:51.586 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:51 smithi103 ceph-mon[32635]: pgmap v825: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:51.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:51 smithi204 ceph-mon[32618]: pgmap v825: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:51.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:51 smithi094 ceph-mon[28050]: pgmap v825: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:53.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:53 smithi204 ceph-mon[32618]: pgmap v826: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:53.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:53 smithi094 ceph-mon[28050]: pgmap v826: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:53.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:53 smithi103 ceph-mon[32635]: pgmap v826: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:24:55.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:55 smithi204 ceph-mon[32618]: pgmap v827: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:55.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:55 smithi094 ceph-mon[28050]: pgmap v827: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:55 smithi103 ceph-mon[32635]: pgmap v827: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:57.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:57 smithi204 ceph-mon[32618]: pgmap v828: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:57.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:57 smithi094 ceph-mon[28050]: pgmap v828: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:57.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:57 smithi103 ceph-mon[32635]: pgmap v828: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:24:59.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:24:59 smithi204 ceph-mon[32618]: pgmap v829: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:59.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:24:59 smithi094 ceph-mon[28050]: pgmap v829: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:24:59.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:24:59 smithi103 ceph-mon[32635]: pgmap v829: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:01.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:01 smithi204 ceph-mon[32618]: pgmap v830: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:01.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:01 smithi094 ceph-mon[28050]: pgmap v830: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:01.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:01 smithi103 ceph-mon[32635]: pgmap v830: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:03.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:03 smithi204 ceph-mon[32618]: pgmap v831: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:03.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:03 smithi094 ceph-mon[28050]: pgmap v831: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:03.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:03 smithi103 ceph-mon[32635]: pgmap v831: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:05.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:05 smithi204 ceph-mon[32618]: pgmap v832: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:05.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:05 smithi094 ceph-mon[28050]: pgmap v832: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:05.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:05 smithi103 ceph-mon[32635]: pgmap v832: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:07.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:07 smithi204 ceph-mon[32618]: pgmap v833: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:07.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:07 smithi094 ceph-mon[28050]: pgmap v833: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:07.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:07 smithi103 ceph-mon[32635]: pgmap v833: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:09.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:09 smithi204 ceph-mon[32618]: pgmap v834: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:09.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:09 smithi094 ceph-mon[28050]: pgmap v834: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:09.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:09 smithi103 ceph-mon[32635]: pgmap v834: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:11.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:11 smithi204 ceph-mon[32618]: pgmap v835: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:11.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:11 smithi094 ceph-mon[28050]: pgmap v835: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:11.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:11 smithi103 ceph-mon[32635]: pgmap v835: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:13.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:13 smithi204 ceph-mon[32618]: pgmap v836: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:13.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:13 smithi094 ceph-mon[28050]: pgmap v836: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:13.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:13 smithi103 ceph-mon[32635]: pgmap v836: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:15.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:15 smithi204 ceph-mon[32618]: pgmap v837: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:15.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:15 smithi094 ceph-mon[28050]: pgmap v837: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:15.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:15 smithi103 ceph-mon[32635]: pgmap v837: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:17.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:17 smithi204 ceph-mon[32618]: pgmap v838: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:17.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:17 smithi094 ceph-mon[28050]: pgmap v838: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:17.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:17 smithi103 ceph-mon[32635]: pgmap v838: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:19.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:19 smithi204 ceph-mon[32618]: pgmap v839: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:19.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:19 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:19.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:19 smithi094 ceph-mon[28050]: pgmap v839: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:19.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:19 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:19.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:19 smithi103 ceph-mon[32635]: pgmap v839: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:19.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:19 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:20.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:20.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:20.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:25:20.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:20.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:20.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:25:20.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:20.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:20.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:25:21.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:21 smithi094 ceph-mon[28050]: pgmap v840: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:22.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:21 smithi103 ceph-mon[32635]: pgmap v840: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:22.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:21 smithi204 ceph-mon[32618]: pgmap v840: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:23.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:23 smithi094 ceph-mon[28050]: pgmap v841: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:24.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:23 smithi103 ceph-mon[32635]: pgmap v841: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:24.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:23 smithi204 ceph-mon[32618]: pgmap v841: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:25.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:25 smithi094 ceph-mon[28050]: pgmap v842: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:26.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:25 smithi103 ceph-mon[32635]: pgmap v842: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:26.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:25 smithi204 ceph-mon[32618]: pgmap v842: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:27.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:27 smithi094 ceph-mon[28050]: pgmap v843: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:28.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:27 smithi103 ceph-mon[32635]: pgmap v843: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:28.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:27 smithi204 ceph-mon[32618]: pgmap v843: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:29.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:29 smithi094 ceph-mon[28050]: pgmap v844: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:30.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:29 smithi103 ceph-mon[32635]: pgmap v844: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:30.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:29 smithi204 ceph-mon[32618]: pgmap v844: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:31.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:31 smithi094 ceph-mon[28050]: pgmap v845: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:32.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:31 smithi103 ceph-mon[32635]: pgmap v845: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:32.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:31 smithi204 ceph-mon[32618]: pgmap v845: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:33.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:33 smithi094 ceph-mon[28050]: pgmap v846: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:34.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:33 smithi103 ceph-mon[32635]: pgmap v846: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:34.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:33 smithi204 ceph-mon[32618]: pgmap v846: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:36.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:35 smithi103 ceph-mon[32635]: pgmap v847: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:36.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:35 smithi204 ceph-mon[32618]: pgmap v847: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:36.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:35 smithi094 ceph-mon[28050]: pgmap v847: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:37 smithi103 ceph-mon[32635]: pgmap v848: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:38.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:37 smithi204 ceph-mon[32618]: pgmap v848: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:38.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:37 smithi094 ceph-mon[28050]: pgmap v848: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:40.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:39 smithi103 ceph-mon[32635]: pgmap v849: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:40.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:39 smithi204 ceph-mon[32618]: pgmap v849: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:40.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:39 smithi094 ceph-mon[28050]: pgmap v849: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:42.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:41 smithi103 ceph-mon[32635]: pgmap v850: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:42.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:41 smithi204 ceph-mon[32618]: pgmap v850: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:42.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:41 smithi094 ceph-mon[28050]: pgmap v850: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:44.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:43 smithi103 ceph-mon[32635]: pgmap v851: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:44.131 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:43 smithi204 ceph-mon[32618]: pgmap v851: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:44.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:43 smithi094 ceph-mon[28050]: pgmap v851: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:46.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:45 smithi094 ceph-mon[28050]: pgmap v852: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:46.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:45 smithi103 ceph-mon[32635]: pgmap v852: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:46.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:45 smithi204 ceph-mon[32618]: pgmap v852: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:48.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:47 smithi094 ceph-mon[28050]: pgmap v853: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:48.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:47 smithi103 ceph-mon[32635]: pgmap v853: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:48.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:47 smithi204 ceph-mon[32618]: pgmap v853: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:50.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:49 smithi094 ceph-mon[28050]: pgmap v854: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:50.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:49 smithi103 ceph-mon[32635]: pgmap v854: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:50.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:49 smithi204 ceph-mon[32618]: pgmap v854: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:52.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:51 smithi094 ceph-mon[28050]: pgmap v855: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:52.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:51 smithi103 ceph-mon[32635]: pgmap v855: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:52.382 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:51 smithi204 ceph-mon[32618]: pgmap v855: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:54.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:53 smithi094 ceph-mon[28050]: pgmap v856: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:54.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:53 smithi103 ceph-mon[32635]: pgmap v856: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:54.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:53 smithi204 ceph-mon[32618]: pgmap v856: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:25:55.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:54 smithi094 ceph-mon[28050]: pgmap v857: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:55.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:54 smithi103 ceph-mon[32635]: pgmap v857: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:55.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:54 smithi204 ceph-mon[32618]: pgmap v857: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:57.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:57 smithi094 ceph-mon[28050]: pgmap v858: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:57.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:57 smithi103 ceph-mon[32635]: pgmap v858: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:57.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:57 smithi204 ceph-mon[32618]: pgmap v858: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:25:59.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:25:59 smithi094 ceph-mon[28050]: pgmap v859: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:59.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:25:59 smithi103 ceph-mon[32635]: pgmap v859: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:25:59.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:25:59 smithi204 ceph-mon[32618]: pgmap v859: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:01.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:01 smithi094 ceph-mon[28050]: pgmap v860: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:01.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:01 smithi103 ceph-mon[32635]: pgmap v860: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:01.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:01 smithi204 ceph-mon[32618]: pgmap v860: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:03.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:03 smithi094 ceph-mon[28050]: pgmap v861: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:03.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:03 smithi103 ceph-mon[32635]: pgmap v861: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:03.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:03 smithi204 ceph-mon[32618]: pgmap v861: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:05.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:05 smithi094 ceph-mon[28050]: pgmap v862: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:05.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:05 smithi103 ceph-mon[32635]: pgmap v862: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:05.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:05 smithi204 ceph-mon[32618]: pgmap v862: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:07.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:07 smithi094 ceph-mon[28050]: pgmap v863: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:07.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:07 smithi103 ceph-mon[32635]: pgmap v863: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:07.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:07 smithi204 ceph-mon[32618]: pgmap v863: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:09.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:09 smithi094 ceph-mon[28050]: pgmap v864: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:09.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:09 smithi103 ceph-mon[32635]: pgmap v864: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:09.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:09 smithi204 ceph-mon[32618]: pgmap v864: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:11.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:11 smithi094 ceph-mon[28050]: pgmap v865: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:11.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:11 smithi103 ceph-mon[32635]: pgmap v865: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:11.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:11 smithi204 ceph-mon[32618]: pgmap v865: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:13.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:13 smithi094 ceph-mon[28050]: pgmap v866: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:13.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:13 smithi103 ceph-mon[32635]: pgmap v866: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:13.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:13 smithi204 ceph-mon[32618]: pgmap v866: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:15.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:15 smithi094 ceph-mon[28050]: pgmap v867: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:15.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:15 smithi103 ceph-mon[32635]: pgmap v867: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:15.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:15 smithi204 ceph-mon[32618]: pgmap v867: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:17.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:17 smithi094 ceph-mon[28050]: pgmap v868: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:17.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:17 smithi103 ceph-mon[32635]: pgmap v868: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:17.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:17 smithi204 ceph-mon[32618]: pgmap v868: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:19.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:19 smithi094 ceph-mon[28050]: pgmap v869: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:19.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:19 smithi103 ceph-mon[32635]: pgmap v869: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:19.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:19 smithi204 ceph-mon[32618]: pgmap v869: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:20.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:20 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:20.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:20 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:20.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:20 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:21.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:21 smithi094 ceph-mon[28050]: pgmap v870: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:21.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:21.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:21 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:21.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:21 smithi103 ceph-mon[32635]: pgmap v870: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:21.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:21.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:21 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:21.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:21 smithi204 ceph-mon[32618]: pgmap v870: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:21.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:21.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:21 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:22.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:26:22.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:26:22.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:26:23.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:23 smithi103 ceph-mon[32635]: pgmap v871: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:23.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:23 smithi204 ceph-mon[32618]: pgmap v871: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:23.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:23 smithi094 ceph-mon[28050]: pgmap v871: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:25.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:25 smithi103 ceph-mon[32635]: pgmap v872: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:25.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:25 smithi204 ceph-mon[32618]: pgmap v872: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:25.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:25 smithi094 ceph-mon[28050]: pgmap v872: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:27.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:27 smithi103 ceph-mon[32635]: pgmap v873: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:27.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:27 smithi204 ceph-mon[32618]: pgmap v873: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:27.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:27 smithi094 ceph-mon[28050]: pgmap v873: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:29.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:29 smithi103 ceph-mon[32635]: pgmap v874: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:29.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:29 smithi204 ceph-mon[32618]: pgmap v874: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:29 smithi094 ceph-mon[28050]: pgmap v874: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:31.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:31 smithi103 ceph-mon[32635]: pgmap v875: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:31.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:31 smithi204 ceph-mon[32618]: pgmap v875: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:31 smithi094 ceph-mon[28050]: pgmap v875: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:33.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:33 smithi103 ceph-mon[32635]: pgmap v876: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:33.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:33 smithi204 ceph-mon[32618]: pgmap v876: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:33.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:33 smithi094 ceph-mon[28050]: pgmap v876: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:35.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:35 smithi103 ceph-mon[32635]: pgmap v877: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:35.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:35 smithi204 ceph-mon[32618]: pgmap v877: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:35.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:35 smithi094 ceph-mon[28050]: pgmap v877: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:37.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:37 smithi204 ceph-mon[32618]: pgmap v878: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:37.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:37 smithi094 ceph-mon[28050]: pgmap v878: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:37.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:37 smithi103 ceph-mon[32635]: pgmap v878: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:39.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:39 smithi204 ceph-mon[32618]: pgmap v879: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:39.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:39 smithi094 ceph-mon[28050]: pgmap v879: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:39.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:39 smithi103 ceph-mon[32635]: pgmap v879: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:41.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:41 smithi204 ceph-mon[32618]: pgmap v880: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:41.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:41 smithi094 ceph-mon[28050]: pgmap v880: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:41 smithi103 ceph-mon[32635]: pgmap v880: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:43.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:43 smithi204 ceph-mon[32618]: pgmap v881: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:43.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:43 smithi094 ceph-mon[28050]: pgmap v881: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:43.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:43 smithi103 ceph-mon[32635]: pgmap v881: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:45.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:45 smithi204 ceph-mon[32618]: pgmap v882: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:45.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:45 smithi094 ceph-mon[28050]: pgmap v882: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:45.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:45 smithi103 ceph-mon[32635]: pgmap v882: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:47.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:47 smithi204 ceph-mon[32618]: pgmap v883: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:47.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:47 smithi094 ceph-mon[28050]: pgmap v883: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:47.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:47 smithi103 ceph-mon[32635]: pgmap v883: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:49.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:49 smithi204 ceph-mon[32618]: pgmap v884: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:49.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:49 smithi094 ceph-mon[28050]: pgmap v884: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:49.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:49 smithi103 ceph-mon[32635]: pgmap v884: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:51.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:51 smithi204 ceph-mon[32618]: pgmap v885: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:51.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:51 smithi094 ceph-mon[28050]: pgmap v885: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:51.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:51 smithi103 ceph-mon[32635]: pgmap v885: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:53.586 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:53 smithi103 ceph-mon[32635]: pgmap v886: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:53.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:53 smithi204 ceph-mon[32618]: pgmap v886: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:53.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:53 smithi094 ceph-mon[28050]: pgmap v886: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:26:55.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:55 smithi204 ceph-mon[32618]: pgmap v887: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:55.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:55 smithi094 ceph-mon[28050]: pgmap v887: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:55.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:55 smithi103 ceph-mon[32635]: pgmap v887: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:57.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:57 smithi204 ceph-mon[32618]: pgmap v888: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:57.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:57 smithi094 ceph-mon[28050]: pgmap v888: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:57.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:57 smithi103 ceph-mon[32635]: pgmap v888: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:26:59.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:26:59 smithi204 ceph-mon[32618]: pgmap v889: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:59.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:26:59 smithi094 ceph-mon[28050]: pgmap v889: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:26:59.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:26:59 smithi103 ceph-mon[32635]: pgmap v889: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:02.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:01 smithi094 ceph-mon[28050]: pgmap v890: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:02.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:01 smithi103 ceph-mon[32635]: pgmap v890: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:02.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:01 smithi204 ceph-mon[32618]: pgmap v890: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:04.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:03 smithi094 ceph-mon[28050]: pgmap v891: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:04.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:03 smithi103 ceph-mon[32635]: pgmap v891: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:04.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:03 smithi204 ceph-mon[32618]: pgmap v891: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:05.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:04 smithi103 ceph-mon[32635]: pgmap v892: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:05.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:04 smithi204 ceph-mon[32618]: pgmap v892: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:05.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:04 smithi094 ceph-mon[28050]: pgmap v892: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:07.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:07 smithi094 ceph-mon[28050]: pgmap v893: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:07.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:07 smithi103 ceph-mon[32635]: pgmap v893: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:07.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:07 smithi204 ceph-mon[32618]: pgmap v893: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:09.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:09 smithi094 ceph-mon[28050]: pgmap v894: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:09.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:09 smithi103 ceph-mon[32635]: pgmap v894: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:09.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:09 smithi204 ceph-mon[32618]: pgmap v894: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:12.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:11 smithi094 ceph-mon[28050]: pgmap v895: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:12.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:11 smithi103 ceph-mon[32635]: pgmap v895: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:12.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:11 smithi204 ceph-mon[32618]: pgmap v895: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:13.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:12 smithi094 ceph-mon[28050]: pgmap v896: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:13.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:12 smithi103 ceph-mon[32635]: pgmap v896: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:13.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:12 smithi204 ceph-mon[32618]: pgmap v896: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:15.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:15 smithi094 ceph-mon[28050]: pgmap v897: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:15.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:15 smithi103 ceph-mon[32635]: pgmap v897: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:15.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:15 smithi204 ceph-mon[32618]: pgmap v897: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:17.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:17 smithi094 ceph-mon[28050]: pgmap v898: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:17.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:17 smithi103 ceph-mon[32635]: pgmap v898: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:17.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:17 smithi204 ceph-mon[32618]: pgmap v898: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:19.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:19 smithi094 ceph-mon[28050]: pgmap v899: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:19.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:19 smithi103 ceph-mon[32635]: pgmap v899: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:19.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:19 smithi204 ceph-mon[32618]: pgmap v899: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:21.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:21 smithi094 ceph-mon[28050]: pgmap v900: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:21.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:21 smithi103 ceph-mon[32635]: pgmap v900: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:21.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:21 smithi204 ceph-mon[32618]: pgmap v900: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:22.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:22 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:22.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:22 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:22.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:22 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: pgmap v901: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.488 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: Adjusting osd_memory_target on smithi094 to 176.4M 2024-04-26T20:27:23.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: Unable to set osd_memory_target on smithi094 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-04-26T20:27:23.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:23.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:23.489 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:23 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:27:23.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: pgmap v901: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:23.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: Adjusting osd_memory_target on smithi094 to 176.4M 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: Unable to set osd_memory_target on smithi094 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:23.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:23 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:27:23.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: pgmap v901: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: Adjusting osd_memory_target on smithi094 to 176.4M 2024-04-26T20:27:23.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: Unable to set osd_memory_target on smithi094 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-04-26T20:27:23.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:23.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:23.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:23 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' 2024-04-26T20:27:25.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:25 smithi094 ceph-mon[28050]: pgmap v902: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:25.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:25 smithi103 ceph-mon[32635]: pgmap v902: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:25.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:25 smithi204 ceph-mon[32618]: pgmap v902: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:27.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:27 smithi094 ceph-mon[28050]: pgmap v903: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:27.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:27 smithi103 ceph-mon[32635]: pgmap v903: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:27.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:27 smithi204 ceph-mon[32618]: pgmap v903: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:29.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:29 smithi103 ceph-mon[32635]: pgmap v904: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:29.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:29 smithi204 ceph-mon[32618]: pgmap v904: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:29.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:29 smithi094 ceph-mon[28050]: pgmap v904: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:31.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:31 smithi103 ceph-mon[32635]: pgmap v905: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:31.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:31 smithi204 ceph-mon[32618]: pgmap v905: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:31.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:31 smithi094 ceph-mon[28050]: pgmap v905: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:33.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:33 smithi103 ceph-mon[32635]: pgmap v906: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:33.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:33 smithi204 ceph-mon[32618]: pgmap v906: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:33.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:33 smithi094 ceph-mon[28050]: pgmap v906: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T20:27:34.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:34.980+0000 7f325ea00640 1 -- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440a3c30 msgr2=0x7f32440a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900009615) for more than 900000000 us, fault. 2024-04-26T20:27:34.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:34.980+0000 7f325ea00640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440a3c30 0x7f32440a60f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3250009230 tx=0x7f32500314a0 comp rx=0 tx=0).stop 2024-04-26T20:27:34.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:34.980+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440a3c30 msgr2=0x7f32440a60f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T20:27:34.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:34.980+0000 7f3257400640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440a3c30 0x7f32440a60f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:34.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:34.980+0000 7f3257400640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440af3e0 0x7f32440b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:34.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:34.980+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f32440b1db0 con 0x7f32440af3e0 2024-04-26T20:27:34.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:34.981+0000 7f325ea00640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440af3e0 0x7f32440b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:34.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:34.983+0000 7f325ea00640 1 --2- 172.21.15.94:0/56252003 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f32440af3e0 0x7f32440b17d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3250009230 tx=0x7f3250041000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:35.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.111+0000 7f3256000640 1 -- 172.21.15.94:0/56252003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f3238006360 con 0x7f326010a990 2024-04-26T20:27:35.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.112+0000 7f3257400640 1 -- 172.21.15.94:0/56252003 <== mon.0 v2:172.21.15.94:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f3238006360 con 0x7f326010a990 2024-04-26T20:27:35.205 INFO:teuthology.orchestra.run.smithi094.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-26T20:27:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-26T20:27:35.229 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-26T20:27:35.480 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c06fdcd85ecc4404925ad110487ee4a0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-26T20:27:35.482 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T20:27:35.491 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T20:27:35.492 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:27:35.492 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:27:35.496 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:35 smithi094 ceph-mon[28050]: pgmap v907: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:35.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:35 smithi103 ceph-mon[32635]: pgmap v907: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:35.554 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-04-26T20:27:35.554 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:27:35.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:27:35.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:35 smithi204 ceph-mon[32618]: pgmap v907: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:35.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.811+0000 7f06b6800640 1 -- 172.21.15.94:0/3464290934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0105d80 msgr2=0x7f06b0106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:35.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.811+0000 7f06b6800640 1 --2- 172.21.15.94:0/3464290934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0105d80 0x7f06b0106200 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0698009900 tx=0x7f069802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:27:35.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.811+0000 7f06b6800640 1 -- 172.21.15.94:0/3464290934 shutdown_connections 2024-04-26T20:27:35.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.811+0000 7f06b6800640 1 --2- 172.21.15.94:0/3464290934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b0106740 0x7f06b010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:35.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.811+0000 7f06b6800640 1 --2- 172.21.15.94:0/3464290934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0105d80 0x7f06b0106200 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:35.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.811+0000 7f06b6800640 1 --2- 172.21.15.94:0/3464290934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b0104b90 0x7f06b0104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:35.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.811+0000 7f06b6800640 1 -- 172.21.15.94:0/3464290934 >> 172.21.15.94:0/3464290934 conn(0x7f06b0100120 msgr2=0x7f06b0102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:35.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.811+0000 7f06b6800640 1 -- 172.21.15.94:0/3464290934 shutdown_connections 2024-04-26T20:27:35.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.811+0000 7f06b6800640 1 -- 172.21.15.94:0/3464290934 wait complete. 2024-04-26T20:27:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.812+0000 7f06b6800640 1 Processor -- start 2024-04-26T20:27:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06b6800640 1 -- start start 2024-04-26T20:27:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06b6800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0104b90 0x7f06b019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:35.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06b6800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b0105d80 0x7f06b019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06b6800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b0106740 0x7f06b01a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06b6800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f06b01106a0 con 0x7f06b0104b90 2024-04-26T20:27:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06b6800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06b0110520 con 0x7f06b0105d80 2024-04-26T20:27:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06b6800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f06b0110820 con 0x7f06b0106740 2024-04-26T20:27:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06af400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0104b90 0x7f06b019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06a6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b0105d80 0x7f06b019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06af400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0104b90 0x7f06b019adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40458/0 (socket says 172.21.15.94:40458) 2024-04-26T20:27:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.813+0000 7f06af400640 1 -- 172.21.15.94:0/681954944 learned_addr learned my addr 172.21.15.94:0/681954944 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:35.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.814+0000 7f06afe00640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b0106740 0x7f06b01a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.814+0000 7f06a6a00640 1 -- 172.21.15.94:0/681954944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b0106740 msgr2=0x7f06b01a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.814+0000 7f06a6a00640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b0106740 0x7f06b01a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.814+0000 7f06a6a00640 1 -- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0104b90 msgr2=0x7f06b019adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.814+0000 7f06a6a00640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0104b90 0x7f06b019adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.814+0000 7f06a6a00640 1 -- 172.21.15.94:0/681954944 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b01a29d0 con 0x7f06b0105d80 2024-04-26T20:27:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.814+0000 7f06af400640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0104b90 0x7f06b019adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:27:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.814+0000 7f06a6a00640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b0105d80 0x7f06b019b300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0698006010 tx=0x7f06980042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:35.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.815+0000 7f06acc00640 1 -- 172.21.15.94:0/681954944 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069803f030 con 0x7f06b0105d80 2024-04-26T20:27:35.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.815+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b01a2c60 con 0x7f06b0105d80 2024-04-26T20:27:35.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.815+0000 7f06acc00640 1 -- 172.21.15.94:0/681954944 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0698031910 con 0x7f06b0105d80 2024-04-26T20:27:35.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.815+0000 7f06acc00640 1 -- 172.21.15.94:0/681954944 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0698035070 con 0x7f06b0105d80 2024-04-26T20:27:35.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.815+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b01a31a0 con 0x7f06b0105d80 2024-04-26T20:27:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.816+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b010bf20 con 0x7f06b0105d80 2024-04-26T20:27:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.818+0000 7f06acc00640 1 -- 172.21.15.94:0/681954944 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0698042030 con 0x7f06b0105d80 2024-04-26T20:27:35.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.819+0000 7f06acc00640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f06940a3cc0 0x7f06940a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.820+0000 7f06acc00640 1 -- 172.21.15.94:0/681954944 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f06980f0ee0 con 0x7f06b0105d80 2024-04-26T20:27:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.820+0000 7f06af400640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f06940a3cc0 0x7f06940a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:35.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.820+0000 7f06af400640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f06940a3cc0 0x7f06940a6180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f06b019c290 tx=0x7f06a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:35.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:35.824+0000 7f06acc00640 1 -- 172.21.15.94:0/681954944 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f06980bcce0 con 0x7f06b0105d80 2024-04-26T20:27:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.027+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06b002dd30 con 0x7f06b0105d80 2024-04-26T20:27:36.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.029+0000 7f06acc00640 1 -- 172.21.15.94:0/681954944 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24879 (secure 0 0 0) 0x7f06980c0fe0 con 0x7f06b0105d80 2024-04-26T20:27:36.030 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:27:36.031 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:10:13.755106+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:27:36.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f06940a3cc0 msgr2=0x7f06940a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f06940a3cc0 0x7f06940a6180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f06b019c290 tx=0x7f06a0005f90 comp rx=0 tx=0).stop 2024-04-26T20:27:36.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b0105d80 msgr2=0x7f06b019b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b0105d80 0x7f06b019b300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0698006010 tx=0x7f06980042f0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 shutdown_connections 2024-04-26T20:27:36.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f06940a3cc0 0x7f06940a6180 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b0106740 0x7f06b01a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b0105d80 0x7f06b019b300 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 --2- 172.21.15.94:0/681954944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06b0104b90 0x7f06b019adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 >> 172.21.15.94:0/681954944 conn(0x7f06b0100120 msgr2=0x7f06b01090f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:36.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 shutdown_connections 2024-04-26T20:27:36.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.032+0000 7f06b6800640 1 -- 172.21.15.94:0/681954944 wait complete. 2024-04-26T20:27:36.054 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:02:53.592939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:08:13.904332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:08:15.670247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:08:23.682553+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:27:36.054 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:27:36.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.263+0000 7f5d71200640 1 -- 172.21.15.94:0/755911037 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6c1010f0 msgr2=0x7f5d6c101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.263+0000 7f5d71200640 1 --2- 172.21.15.94:0/755911037 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6c1010f0 0x7f5d6c101510 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009ca0 tx=0x7f5d5c02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.263+0000 7f5d71200640 1 -- 172.21.15.94:0/755911037 shutdown_connections 2024-04-26T20:27:36.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.263+0000 7f5d71200640 1 --2- 172.21.15.94:0/755911037 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c1023f0 0x7f5d6c102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.263+0000 7f5d71200640 1 --2- 172.21.15.94:0/755911037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6c101a50 0x7f5d6c101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.263+0000 7f5d71200640 1 --2- 172.21.15.94:0/755911037 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6c1010f0 0x7f5d6c101510 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.263+0000 7f5d71200640 1 -- 172.21.15.94:0/755911037 >> 172.21.15.94:0/755911037 conn(0x7f5d6c0fccd0 msgr2=0x7f5d6c0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:36.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.263+0000 7f5d71200640 1 -- 172.21.15.94:0/755911037 shutdown_connections 2024-04-26T20:27:36.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.263+0000 7f5d71200640 1 -- 172.21.15.94:0/755911037 wait complete. 2024-04-26T20:27:36.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.264+0000 7f5d71200640 1 Processor -- start 2024-04-26T20:27:36.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d71200640 1 -- start start 2024-04-26T20:27:36.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d71200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6c1010f0 0x7f5d6c19ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d71200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6c101a50 0x7f5d6c19b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d71200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c1023f0 0x7f5d6c1a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d71200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c1104f0 con 0x7f5d6c1023f0 2024-04-26T20:27:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d71200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c110370 con 0x7f5d6c1010f0 2024-04-26T20:27:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d71200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c110670 con 0x7f5d6c101a50 2024-04-26T20:27:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d6b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6c1010f0 0x7f5d6c19ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d6b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6c1010f0 0x7f5d6c19ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:33700/0 (socket says 172.21.15.94:33700) 2024-04-26T20:27:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d6b400640 1 -- 172.21.15.94:0/2902341635 learned_addr learned my addr 172.21.15.94:0/2902341635 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:36.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d6be00640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c1023f0 0x7f5d6c1a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:36.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.265+0000 7f5d62a00640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6c101a50 0x7f5d6c19b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:36.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.266+0000 7f5d6b400640 1 -- 172.21.15.94:0/2902341635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6c101a50 msgr2=0x7f5d6c19b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.266+0000 7f5d6b400640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6c101a50 0x7f5d6c19b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.266+0000 7f5d6b400640 1 -- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c1023f0 msgr2=0x7f5d6c1a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.266+0000 7f5d6b400640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c1023f0 0x7f5d6c1a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.266+0000 7f5d6b400640 1 -- 172.21.15.94:0/2902341635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d6c1a2830 con 0x7f5d6c1010f0 2024-04-26T20:27:36.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.266+0000 7f5d6b400640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6c1010f0 0x7f5d6c19ac20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c031a20 tx=0x7f5d5c003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:36.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.266+0000 7f5d68c00640 1 -- 172.21.15.94:0/2902341635 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c0317c0 con 0x7f5d6c1010f0 2024-04-26T20:27:36.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.267+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d6c1a2ac0 con 0x7f5d6c1010f0 2024-04-26T20:27:36.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.267+0000 7f5d68c00640 1 -- 172.21.15.94:0/2902341635 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d5c0044f0 con 0x7f5d6c1010f0 2024-04-26T20:27:36.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.267+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d6c1a3060 con 0x7f5d6c1010f0 2024-04-26T20:27:36.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.267+0000 7f5d68c00640 1 -- 172.21.15.94:0/2902341635 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c02fc10 con 0x7f5d6c1010f0 2024-04-26T20:27:36.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.268+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d6c10be40 con 0x7f5d6c1010f0 2024-04-26T20:27:36.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.269+0000 7f5d68c00640 1 -- 172.21.15.94:0/2902341635 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5d5c004040 con 0x7f5d6c1010f0 2024-04-26T20:27:36.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.271+0000 7f5d68c00640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5d500a3cc0 0x7f5d500a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:36.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.271+0000 7f5d68c00640 1 -- 172.21.15.94:0/2902341635 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f5d5c0f1eb0 con 0x7f5d6c1010f0 2024-04-26T20:27:36.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.271+0000 7f5d62a00640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5d500a3cc0 0x7f5d500a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:36.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.272+0000 7f5d62a00640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5d500a3cc0 0x7f5d500a6180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009bd0 tx=0x7f5d54005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:36.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.275+0000 7f5d68c00640 1 -- 172.21.15.94:0/2902341635 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5d5c0be410 con 0x7f5d6c1010f0 2024-04-26T20:27:36.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.474+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5d6c02dcc0 con 0x7f5d6c1010f0 2024-04-26T20:27:36.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.475+0000 7f5d68c00640 1 -- 172.21.15.94:0/2902341635 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f5d5c0bdd10 con 0x7f5d6c1010f0 2024-04-26T20:27:36.477 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-04-26T20:27:36.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5d500a3cc0 msgr2=0x7f5d500a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5d500a3cc0 0x7f5d500a6180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009bd0 tx=0x7f5d54005f90 comp rx=0 tx=0).stop 2024-04-26T20:27:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6c1010f0 msgr2=0x7f5d6c19ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6c1010f0 0x7f5d6c19ac20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c031a20 tx=0x7f5d5c003ba0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 shutdown_connections 2024-04-26T20:27:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f5d500a3cc0 0x7f5d500a6180 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c1023f0 0x7f5d6c1a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6c101a50 0x7f5d6c19b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 --2- 172.21.15.94:0/2902341635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d6c1010f0 0x7f5d6c19ac20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 >> 172.21.15.94:0/2902341635 conn(0x7f5d6c0fccd0 msgr2=0x7f5d6c100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 shutdown_connections 2024-04-26T20:27:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.478+0000 7f5d71200640 1 -- 172.21.15.94:0/2902341635 wait complete. 2024-04-26T20:27:36.486 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:36 smithi094 ceph-mon[28050]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:27:36.487 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:36 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/681954944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:27:36.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:27:36.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:36 smithi103 ceph-mon[32635]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:27:36.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:36 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/681954944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:27:36.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:36 smithi204 ceph-mon[32618]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:27:36.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:36 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/681954944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:27:36.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.705+0000 7f3a8ea00640 1 -- 172.21.15.94:0/1193766522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a88109fc0 msgr2=0x7f3a8810a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.705+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/1193766522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a88109fc0 0x7f3a8810a440 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009900 tx=0x7f3a7402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.705+0000 7f3a8ea00640 1 -- 172.21.15.94:0/1193766522 shutdown_connections 2024-04-26T20:27:36.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.705+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/1193766522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a8810a980 0x7f3a881112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.705+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/1193766522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a88109fc0 0x7f3a8810a440 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.705+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/1193766522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a88108e70 0x7f3a88109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.705+0000 7f3a8ea00640 1 -- 172.21.15.94:0/1193766522 >> 172.21.15.94:0/1193766522 conn(0x7f3a88077e00 msgr2=0x7f3a8807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:36.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.705+0000 7f3a8ea00640 1 -- 172.21.15.94:0/1193766522 shutdown_connections 2024-04-26T20:27:36.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.705+0000 7f3a8ea00640 1 -- 172.21.15.94:0/1193766522 wait complete. 2024-04-26T20:27:36.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.706+0000 7f3a8ea00640 1 Processor -- start 2024-04-26T20:27:36.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8ea00640 1 -- start start 2024-04-26T20:27:36.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a88108e70 0x7f3a8819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:36.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a88109fc0 0x7f3a8819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8ea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a8810a980 0x7f3a881a6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8ea00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3a88114a10 con 0x7f3a8810a980 2024-04-26T20:27:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3a88114890 con 0x7f3a88108e70 2024-04-26T20:27:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8ea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3a88114b90 con 0x7f3a88109fc0 2024-04-26T20:27:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8e000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a8810a980 0x7f3a881a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8e000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a8810a980 0x7f3a881a6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40490/0 (socket says 172.21.15.94:40490) 2024-04-26T20:27:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8e000640 1 -- 172.21.15.94:0/3960969399 learned_addr learned my addr 172.21.15.94:0/3960969399 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8cc00640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a88109fc0 0x7f3a8819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.707+0000 7f3a8d600640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a88108e70 0x7f3a8819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a8e000640 1 -- 172.21.15.94:0/3960969399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a88109fc0 msgr2=0x7f3a8819f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a8e000640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a88109fc0 0x7f3a8819f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a8e000640 1 -- 172.21.15.94:0/3960969399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a88108e70 msgr2=0x7f3a8819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a8e000640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a88108e70 0x7f3a8819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a8e000640 1 -- 172.21.15.94:0/3960969399 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a881a6cd0 con 0x7f3a8810a980 2024-04-26T20:27:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a8e000640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a8810a980 0x7f3a881a6660 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000efd0 tx=0x7f3a8000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a86000640 1 -- 172.21.15.94:0/3960969399 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80003b70 con 0x7f3a8810a980 2024-04-26T20:27:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a86000640 1 -- 172.21.15.94:0/3960969399 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a80003d10 con 0x7f3a8810a980 2024-04-26T20:27:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a881a6fc0 con 0x7f3a8810a980 2024-04-26T20:27:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.708+0000 7f3a86000640 1 -- 172.21.15.94:0/3960969399 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a800108a0 con 0x7f3a8810a980 2024-04-26T20:27:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.709+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a881a7430 con 0x7f3a8810a980 2024-04-26T20:27:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.710+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a88110180 con 0x7f3a8810a980 2024-04-26T20:27:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.712+0000 7f3a86000640 1 -- 172.21.15.94:0/3960969399 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3a80004020 con 0x7f3a8810a980 2024-04-26T20:27:36.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.713+0000 7f3a86000640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a700a3c40 0x7f3a700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:36.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.713+0000 7f3a86000640 1 -- 172.21.15.94:0/3960969399 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f3a800d0da0 con 0x7f3a8810a980 2024-04-26T20:27:36.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.714+0000 7f3a8d600640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a700a3c40 0x7f3a700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:36.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.714+0000 7f3a8d600640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a700a3c40 0x7f3a700a6100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009bd0 tx=0x7f3a7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.719+0000 7f3a86000640 1 -- 172.21.15.94:0/3960969399 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3a8009d1f0 con 0x7f3a8810a980 2024-04-26T20:27:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.920+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3a8802dcd0 con 0x7f3a8810a980 2024-04-26T20:27:36.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.921+0000 7f3a86000640 1 -- 172.21.15.94:0/3960969399 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f3a8009caf0 con 0x7f3a8810a980 2024-04-26T20:27:36.922 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:27:36.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.923+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a700a3c40 msgr2=0x7f3a700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.923+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a700a3c40 0x7f3a700a6100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009bd0 tx=0x7f3a7c005f90 comp rx=0 tx=0).stop 2024-04-26T20:27:36.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.923+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a8810a980 msgr2=0x7f3a881a6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:36.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.923+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a8810a980 0x7f3a881a6660 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000efd0 tx=0x7f3a8000c6a0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.924+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 shutdown_connections 2024-04-26T20:27:36.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.924+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a700a3c40 0x7f3a700a6100 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.924+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a8810a980 0x7f3a881a6660 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.924+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a88109fc0 0x7f3a8819f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.924+0000 7f3a8ea00640 1 --2- 172.21.15.94:0/3960969399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a88108e70 0x7f3a8819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:36.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.924+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 >> 172.21.15.94:0/3960969399 conn(0x7f3a88077e00 msgr2=0x7f3a8810d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:36.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.924+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 shutdown_connections 2024-04-26T20:27:36.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:36.924+0000 7f3a8ea00640 1 -- 172.21.15.94:0/3960969399 wait complete. 2024-04-26T20:27:36.944 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:27:37.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.144+0000 7f0225000640 1 -- 172.21.15.94:0/4286971607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0220104b70 msgr2=0x7f0220104f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.144+0000 7f0225000640 1 --2- 172.21.15.94:0/4286971607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0220104b70 0x7f0220104f90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0210009900 tx=0x7f021002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.145+0000 7f0225000640 1 -- 172.21.15.94:0/4286971607 shutdown_connections 2024-04-26T20:27:37.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.145+0000 7f0225000640 1 --2- 172.21.15.94:0/4286971607 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0220106720 0x7f022010cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.145+0000 7f0225000640 1 --2- 172.21.15.94:0/4286971607 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0220105d60 0x7f02201061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.145+0000 7f0225000640 1 --2- 172.21.15.94:0/4286971607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0220104b70 0x7f0220104f90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.145+0000 7f0225000640 1 -- 172.21.15.94:0/4286971607 >> 172.21.15.94:0/4286971607 conn(0x7f0220100120 msgr2=0x7f0220102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:37.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.145+0000 7f0225000640 1 -- 172.21.15.94:0/4286971607 shutdown_connections 2024-04-26T20:27:37.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.145+0000 7f0225000640 1 -- 172.21.15.94:0/4286971607 wait complete. 2024-04-26T20:27:37.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.146+0000 7f0225000640 1 Processor -- start 2024-04-26T20:27:37.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.146+0000 7f0225000640 1 -- start start 2024-04-26T20:27:37.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.146+0000 7f0225000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0220104b70 0x7f022019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:37.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.146+0000 7f0225000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0220105d60 0x7f022019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:37.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f0225000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0220106720 0x7f02201a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:37.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f0225000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f0220110670 con 0x7f0220106720 2024-04-26T20:27:37.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f0225000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f02201104f0 con 0x7f0220105d60 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f0225000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f02201107f0 con 0x7f0220104b70 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f021e800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0220106720 0x7f02201a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f021e800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0220106720 0x7f02201a2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40506/0 (socket says 172.21.15.94:40506) 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f021e800640 1 -- 172.21.15.94:0/2750777299 learned_addr learned my addr 172.21.15.94:0/2750777299 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f0217e00640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0220105d60 0x7f022019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f021de00640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0220104b70 0x7f022019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f021e800640 1 -- 172.21.15.94:0/2750777299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0220104b70 msgr2=0x7f022019adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f021e800640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0220104b70 0x7f022019adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f021e800640 1 -- 172.21.15.94:0/2750777299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0220105d60 msgr2=0x7f022019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f021e800640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0220105d60 0x7f022019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.147+0000 7f021e800640 1 -- 172.21.15.94:0/2750777299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02201a29d0 con 0x7f0220106720 2024-04-26T20:27:37.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.148+0000 7f021e800640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0220106720 0x7f02201a2330 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f021800eb10 tx=0x7f021800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:37.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.148+0000 7f0216a00640 1 -- 172.21.15.94:0/2750777299 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218004b20 con 0x7f0220106720 2024-04-26T20:27:37.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.148+0000 7f0216a00640 1 -- 172.21.15.94:0/2750777299 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0218004cc0 con 0x7f0220106720 2024-04-26T20:27:37.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.148+0000 7f0216a00640 1 -- 172.21.15.94:0/2750777299 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218010670 con 0x7f0220106720 2024-04-26T20:27:37.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.148+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02201a2cc0 con 0x7f0220106720 2024-04-26T20:27:37.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.148+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02201a31c0 con 0x7f0220106720 2024-04-26T20:27:37.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.149+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f022010bf00 con 0x7f0220106720 2024-04-26T20:27:37.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.151+0000 7f0216a00640 1 -- 172.21.15.94:0/2750777299 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0218005020 con 0x7f0220106720 2024-04-26T20:27:37.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.153+0000 7f0216a00640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02040a3c40 0x7f02040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:37.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.154+0000 7f0216a00640 1 -- 172.21.15.94:0/2750777299 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f02180d10b0 con 0x7f0220106720 2024-04-26T20:27:37.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.154+0000 7f021de00640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02040a3c40 0x7f02040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:37.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.154+0000 7f021de00640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02040a3c40 0x7f02040a6100 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0210009230 tx=0x7f02100314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:37.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.158+0000 7f0216a00640 1 -- 172.21.15.94:0/2750777299 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f021809d500 con 0x7f0220106720 2024-04-26T20:27:37.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.363+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f022002dd20 con 0x7f0220106720 2024-04-26T20:27:37.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.363+0000 7f0216a00640 1 -- 172.21.15.94:0/2750777299 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f021809ce00 con 0x7f0220106720 2024-04-26T20:27:37.365 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-04-26T20:27:37.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02040a3c40 msgr2=0x7f02040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02040a3c40 0x7f02040a6100 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0210009230 tx=0x7f02100314a0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0220106720 msgr2=0x7f02201a2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0220106720 0x7f02201a2330 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f021800eb10 tx=0x7f021800c2d0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 shutdown_connections 2024-04-26T20:27:37.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f02040a3c40 0x7f02040a6100 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0220106720 0x7f02201a2330 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0220105d60 0x7f022019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 --2- 172.21.15.94:0/2750777299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0220104b70 0x7f022019adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 >> 172.21.15.94:0/2750777299 conn(0x7f0220100120 msgr2=0x7f02201090d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:37.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.366+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 shutdown_connections 2024-04-26T20:27:37.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.367+0000 7f0225000640 1 -- 172.21.15.94:0/2750777299 wait complete. 2024-04-26T20:27:37.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:27:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:37 smithi103 ceph-mon[32635]: pgmap v908: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:37 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2902341635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:27:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:37 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3960969399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:27:37.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.593+0000 7f3a65200640 1 -- 172.21.15.94:0/3261678730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 msgr2=0x7f3a60074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.593+0000 7f3a65200640 1 --2- 172.21.15.94:0/3261678730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 0x7f3a60074dc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009900 tx=0x7f3a5002a410 comp rx=0 tx=0).stop 2024-04-26T20:27:37.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.593+0000 7f3a65200640 1 -- 172.21.15.94:0/3261678730 shutdown_connections 2024-04-26T20:27:37.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.593+0000 7f3a65200640 1 --2- 172.21.15.94:0/3261678730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a6010a780 0x7f3a6010cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.593+0000 7f3a65200640 1 --2- 172.21.15.94:0/3261678730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a60075390 0x7f3a60075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.593+0000 7f3a65200640 1 --2- 172.21.15.94:0/3261678730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 0x7f3a60074dc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.593+0000 7f3a65200640 1 -- 172.21.15.94:0/3261678730 >> 172.21.15.94:0/3261678730 conn(0x7f3a600fde30 msgr2=0x7f3a60100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:37.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.593+0000 7f3a65200640 1 -- 172.21.15.94:0/3261678730 shutdown_connections 2024-04-26T20:27:37.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.593+0000 7f3a65200640 1 -- 172.21.15.94:0/3261678730 wait complete. 2024-04-26T20:27:37.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.594+0000 7f3a65200640 1 Processor -- start 2024-04-26T20:27:37.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.594+0000 7f3a65200640 1 -- start start 2024-04-26T20:27:37.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a65200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a60075390 0x7f3a6019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:37.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a65200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 0x7f3a6019b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:37.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a65200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a6010a780 0x7f3a601a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:37.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a65200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f3a60110560 con 0x7f3a60076970 2024-04-26T20:27:37.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a65200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3a601103e0 con 0x7f3a6010a780 2024-04-26T20:27:37.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a65200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3a601106e0 con 0x7f3a60075390 2024-04-26T20:27:37.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5ea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 0x7f3a6019b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:37.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5ea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 0x7f3a6019b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40518/0 (socket says 172.21.15.94:40518) 2024-04-26T20:27:37.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5ea00640 1 -- 172.21.15.94:0/3556051460 learned_addr learned my addr 172.21.15.94:0/3556051460 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:37.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5fe00640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a6010a780 0x7f3a601a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:37.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5f400640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a60075390 0x7f3a6019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:37.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5ea00640 1 -- 172.21.15.94:0/3556051460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a60075390 msgr2=0x7f3a6019ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5ea00640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a60075390 0x7f3a6019ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5ea00640 1 -- 172.21.15.94:0/3556051460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a6010a780 msgr2=0x7f3a601a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5ea00640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a6010a780 0x7f3a601a2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.595+0000 7f3a5ea00640 1 -- 172.21.15.94:0/3556051460 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a601a2910 con 0x7f3a60076970 2024-04-26T20:27:37.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.596+0000 7f3a5ea00640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 0x7f3a6019b240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800eb90 tx=0x7f3a4800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:37.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.596+0000 7f3a57e00640 1 -- 172.21.15.94:0/3556051460 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48004d20 con 0x7f3a60076970 2024-04-26T20:27:37.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.596+0000 7f3a57e00640 1 -- 172.21.15.94:0/3556051460 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a4800ee90 con 0x7f3a60076970 2024-04-26T20:27:37.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.596+0000 7f3a57e00640 1 -- 172.21.15.94:0/3556051460 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48010670 con 0x7f3a60076970 2024-04-26T20:27:37.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.596+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a601a2c00 con 0x7f3a60076970 2024-04-26T20:27:37.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.596+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a601a3100 con 0x7f3a60076970 2024-04-26T20:27:37.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.597+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a60069670 con 0x7f3a60076970 2024-04-26T20:27:37.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.599+0000 7f3a57e00640 1 -- 172.21.15.94:0/3556051460 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3a48005020 con 0x7f3a60076970 2024-04-26T20:27:37.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.601+0000 7f3a57e00640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a400a3c40 0x7f3a400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:37.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.601+0000 7f3a57e00640 1 -- 172.21.15.94:0/3556051460 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f3a480d1070 con 0x7f3a60076970 2024-04-26T20:27:37.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.601+0000 7f3a5f400640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a400a3c40 0x7f3a400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:37.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.602+0000 7f3a5f400640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a400a3c40 0x7f3a400a6100 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009230 tx=0x7f3a500314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:37.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.605+0000 7f3a57e00640 1 -- 172.21.15.94:0/3556051460 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3a4809d4c0 con 0x7f3a60076970 2024-04-26T20:27:37.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:37 smithi204 ceph-mon[32618]: pgmap v908: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:37.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:37 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2902341635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:27:37.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:37 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3960969399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:27:37.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:37 smithi094 ceph-mon[28050]: pgmap v908: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T20:27:37.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:37 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2902341635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:27:37.737 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:37 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3960969399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:27:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.819+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3a6002dad0 con 0x7f3a60076970 2024-04-26T20:27:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.820+0000 7f3a57e00640 1 -- 172.21.15.94:0/3556051460 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f3a4809cdc0 con 0x7f3a60076970 2024-04-26T20:27:37.821 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-04-26T20:27:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a400a3c40 msgr2=0x7f3a400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a400a3c40 0x7f3a400a6100 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009230 tx=0x7f3a500314a0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 msgr2=0x7f3a6019b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 0x7f3a6019b240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800eb90 tx=0x7f3a4800c6a0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 shutdown_connections 2024-04-26T20:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f3a400a3c40 0x7f3a400a6100 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a6010a780 0x7f3a601a2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a60076970 0x7f3a6019b240 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 --2- 172.21.15.94:0/3556051460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a60075390 0x7f3a6019ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 >> 172.21.15.94:0/3556051460 conn(0x7f3a600fde30 msgr2=0x7f3a60106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 shutdown_connections 2024-04-26T20:27:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:37.823+0000 7f3a65200640 1 -- 172.21.15.94:0/3556051460 wait complete. 2024-04-26T20:27:37.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T20:27:38.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.049+0000 7f721c200640 1 -- 172.21.15.94:0/3653054374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7214108fb0 msgr2=0x7f72141093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:38.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.049+0000 7f721c200640 1 --2- 172.21.15.94:0/3653054374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7214108fb0 0x7f72141093d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7208009860 tx=0x7f720802a340 comp rx=0 tx=0).stop 2024-04-26T20:27:38.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.049+0000 7f721c200640 1 -- 172.21.15.94:0/3653054374 shutdown_connections 2024-04-26T20:27:38.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.049+0000 7f721c200640 1 --2- 172.21.15.94:0/3653054374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f721410ab60 0x7f72141113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.049+0000 7f721c200640 1 --2- 172.21.15.94:0/3653054374 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f721410a1a0 0x7f721410a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.049+0000 7f721c200640 1 --2- 172.21.15.94:0/3653054374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7214108fb0 0x7f72141093d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.049+0000 7f721c200640 1 -- 172.21.15.94:0/3653054374 >> 172.21.15.94:0/3653054374 conn(0x7f7214077f40 msgr2=0x7f721407a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:38.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.049+0000 7f721c200640 1 -- 172.21.15.94:0/3653054374 shutdown_connections 2024-04-26T20:27:38.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.049+0000 7f721c200640 1 -- 172.21.15.94:0/3653054374 wait complete. 2024-04-26T20:27:38.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.050+0000 7f721c200640 1 Processor -- start 2024-04-26T20:27:38.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.050+0000 7f721c200640 1 -- start start 2024-04-26T20:27:38.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f721c200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7214108fb0 0x7f721419f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:38.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f721c200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f721410a1a0 0x7f721419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:38.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f721c200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f721410ab60 0x7f72141a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:38.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f721c200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f7214114af0 con 0x7f721410a1a0 2024-04-26T20:27:38.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f721c200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7214114970 con 0x7f721410ab60 2024-04-26T20:27:38.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f721c200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7214114c70 con 0x7f7214108fb0 2024-04-26T20:27:38.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f7218e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f721410a1a0 0x7f721419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:38.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f7218e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f721410a1a0 0x7f721419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40526/0 (socket says 172.21.15.94:40526) 2024-04-26T20:27:38.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f7218e00640 1 -- 172.21.15.94:0/2345012636 learned_addr learned my addr 172.21.15.94:0/2345012636 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:38.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f7219800640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7214108fb0 0x7f721419f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:38.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f721a200640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f721410ab60 0x7f72141a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:38.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f7218e00640 1 -- 172.21.15.94:0/2345012636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7214108fb0 msgr2=0x7f721419f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:38.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f7218e00640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7214108fb0 0x7f721419f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f7218e00640 1 -- 172.21.15.94:0/2345012636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f721410ab60 msgr2=0x7f72141a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f7218e00640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f721410ab60 0x7f72141a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.051+0000 7f7218e00640 1 -- 172.21.15.94:0/2345012636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72141a6dc0 con 0x7f721410a1a0 2024-04-26T20:27:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.052+0000 7f7218e00640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f721410a1a0 0x7f721419f6e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f720000e9c0 tx=0x7f720000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:38.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.052+0000 7f7212000640 1 -- 172.21.15.94:0/2345012636 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720000cde0 con 0x7f721410a1a0 2024-04-26T20:27:38.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.052+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72141a70b0 con 0x7f721410a1a0 2024-04-26T20:27:38.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.052+0000 7f7212000640 1 -- 172.21.15.94:0/2345012636 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72000035f0 con 0x7f721410a1a0 2024-04-26T20:27:38.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.052+0000 7f7212000640 1 -- 172.21.15.94:0/2345012636 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200018630 con 0x7f721410a1a0 2024-04-26T20:27:38.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.052+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72141a7650 con 0x7f721410a1a0 2024-04-26T20:27:38.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.054+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72141103a0 con 0x7f721410a1a0 2024-04-26T20:27:38.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.055+0000 7f7212000640 1 -- 172.21.15.94:0/2345012636 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7200004000 con 0x7f721410a1a0 2024-04-26T20:27:38.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.057+0000 7f7212000640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f71fc0a3c40 0x7f71fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:38.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.057+0000 7f7212000640 1 -- 172.21.15.94:0/2345012636 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f72000d0c00 con 0x7f721410a1a0 2024-04-26T20:27:38.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.057+0000 7f7219800640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f71fc0a3c40 0x7f71fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:38.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.058+0000 7f7219800640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f71fc0a3c40 0x7f71fc0a6100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f720802a930 tx=0x7f7208030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:38.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.061+0000 7f7212000640 1 -- 172.21.15.94:0/2345012636 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f720009d050 con 0x7f721410a1a0 2024-04-26T20:27:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.278+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f721402dad0 con 0x7f721410a1a0 2024-04-26T20:27:38.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.278+0000 7f7212000640 1 -- 172.21.15.94:0/2345012636 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v96) ==== 81+0+591 (secure 0 0 0) 0x7f7200009510 con 0x7f721410a1a0 2024-04-26T20:27:38.280 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3877259400 2024-04-27T20:00:11.413271+0000 2024-04-26T20:27:38.280 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/3361956945 2024-04-27T20:00:11.413271+0000 2024-04-26T20:27:38.280 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3651438916 2024-04-27T20:00:11.413271+0000 2024-04-26T20:27:38.280 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6801/3005317891 2024-04-27T19:59:38.717857+0000 2024-04-26T20:27:38.280 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1688404742 2024-04-27T19:59:38.717857+0000 2024-04-26T20:27:38.281 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/2041870658 2024-04-27T19:59:38.717857+0000 2024-04-26T20:27:38.281 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/442392134 2024-04-27T19:59:38.717857+0000 2024-04-26T20:27:38.281 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:0/1853030572 2024-04-27T20:00:11.413271+0000 2024-04-26T20:27:38.281 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3005317891 2024-04-27T19:59:38.717857+0000 2024-04-26T20:27:38.281 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94:6800/3651438916 2024-04-27T20:00:11.413271+0000 2024-04-26T20:27:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:listed 10 entries 2024-04-26T20:27:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f71fc0a3c40 msgr2=0x7f71fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f71fc0a3c40 0x7f71fc0a6100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f720802a930 tx=0x7f7208030470 comp rx=0 tx=0).stop 2024-04-26T20:27:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f721410a1a0 msgr2=0x7f721419f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f721410a1a0 0x7f721419f6e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f720000e9c0 tx=0x7f720000ee80 comp rx=0 tx=0).stop 2024-04-26T20:27:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 shutdown_connections 2024-04-26T20:27:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f71fc0a3c40 0x7f71fc0a6100 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f721410ab60 0x7f72141a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f721410a1a0 0x7f721419f6e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 --2- 172.21.15.94:0/2345012636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7214108fb0 0x7f721419f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 >> 172.21.15.94:0/2345012636 conn(0x7f7214077f40 msgr2=0x7f721410d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 shutdown_connections 2024-04-26T20:27:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.281+0000 7f721c200640 1 -- 172.21.15.94:0/2345012636 wait complete. 2024-04-26T20:27:38.302 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T20:27:38.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:27:38.302 DEBUG:teuthology.orchestra.run.smithi094:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T20:27:38.398 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi094.front.sepia.ceph.com 2024-04-26T20:27:38.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:27:38.398 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T20:27:38.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:27:38.447 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-04-26T20:27:38.491 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T20:27:38.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:27:38.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T20:27:38.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:38 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2750777299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:27:38.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:38 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3556051460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:27:38.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:38 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2750777299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:27:38.603 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:38 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3556051460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:27:38.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:27:38.619 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2024-04-26T20:27:38.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:38 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2750777299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:27:38.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:38 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3556051460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:27:38.724 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T20:27:38.734 INFO:tasks.cephadm:Teardown begin 2024-04-26T20:27:38.735 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T20:27:38.736 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:27:38.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.916+0000 7fbc58e00640 1 -- 172.21.15.94:0/2167331394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc5410a180 msgr2=0x7fbc5410a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:38.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.916+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2167331394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc5410a180 0x7fbc5410a600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbc44009900 tx=0x7fbc4402a400 comp rx=0 tx=0).stop 2024-04-26T20:27:38.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.916+0000 7fbc58e00640 1 -- 172.21.15.94:0/2167331394 shutdown_connections 2024-04-26T20:27:38.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.916+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2167331394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc5410ab40 0x7fbc541113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.916+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2167331394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc5410a180 0x7fbc5410a600 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.916+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2167331394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc54108f90 0x7fbc541093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.916+0000 7fbc58e00640 1 -- 172.21.15.94:0/2167331394 >> 172.21.15.94:0/2167331394 conn(0x7fbc54077f40 msgr2=0x7fbc5407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:38.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.916+0000 7fbc58e00640 1 -- 172.21.15.94:0/2167331394 shutdown_connections 2024-04-26T20:27:38.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.916+0000 7fbc58e00640 1 -- 172.21.15.94:0/2167331394 wait complete. 2024-04-26T20:27:38.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.917+0000 7fbc58e00640 1 Processor -- start 2024-04-26T20:27:38.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.917+0000 7fbc58e00640 1 -- start start 2024-04-26T20:27:38.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.917+0000 7fbc58e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc54108f90 0x7fbc5419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:38.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.917+0000 7fbc58e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc5410a180 0x7fbc5419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:38.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.917+0000 7fbc58e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc5410ab40 0x7fbc541a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:38.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.917+0000 7fbc58e00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fbc54114b20 con 0x7fbc5410ab40 2024-04-26T20:27:38.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.917+0000 7fbc58e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc541149a0 con 0x7fbc5410a180 2024-04-26T20:27:38.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.917+0000 7fbc58e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc54114ca0 con 0x7fbc54108f90 2024-04-26T20:27:38.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc51e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc54108f90 0x7fbc5419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:38.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc51e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc54108f90 0x7fbc5419f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.94:35004/0 (socket says 172.21.15.94:35004) 2024-04-26T20:27:38.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc51e00640 1 -- 172.21.15.94:0/2147407507 learned_addr learned my addr 172.21.15.94:0/2147407507 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:38.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc52800640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc5410ab40 0x7fbc541a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:38.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc51400640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc5410a180 0x7fbc5419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:38.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc52800640 1 -- 172.21.15.94:0/2147407507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc54108f90 msgr2=0x7fbc5419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:38.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc52800640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc54108f90 0x7fbc5419f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc52800640 1 -- 172.21.15.94:0/2147407507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc5410a180 msgr2=0x7fbc5419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:38.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc52800640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc5410a180 0x7fbc5419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:38.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc52800640 1 -- 172.21.15.94:0/2147407507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc541a6d70 con 0x7fbc5410ab40 2024-04-26T20:27:38.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc51e00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc54108f90 0x7fbc5419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:27:38.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.918+0000 7fbc52800640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc5410ab40 0x7fbc541a6700 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00e940 tx=0x7fbc4c00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:38.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.919+0000 7fbc4aa00640 1 -- 172.21.15.94:0/2147407507 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c004d20 con 0x7fbc5410ab40 2024-04-26T20:27:38.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.919+0000 7fbc4aa00640 1 -- 172.21.15.94:0/2147407507 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc4c005020 con 0x7fbc5410ab40 2024-04-26T20:27:38.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.919+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc541a7060 con 0x7fbc5410ab40 2024-04-26T20:27:38.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.919+0000 7fbc4aa00640 1 -- 172.21.15.94:0/2147407507 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c0116d0 con 0x7fbc5410ab40 2024-04-26T20:27:38.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.919+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc541a7530 con 0x7fbc5410ab40 2024-04-26T20:27:38.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.920+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc541102b0 con 0x7fbc5410ab40 2024-04-26T20:27:38.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.922+0000 7fbc4aa00640 1 -- 172.21.15.94:0/2147407507 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbc4c005460 con 0x7fbc5410ab40 2024-04-26T20:27:38.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.923+0000 7fbc4aa00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbc380a3c40 0x7fbc380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:38.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.923+0000 7fbc4aa00640 1 -- 172.21.15.94:0/2147407507 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fbc4c0d0a50 con 0x7fbc5410ab40 2024-04-26T20:27:38.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.924+0000 7fbc51e00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbc380a3c40 0x7fbc380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:38.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.924+0000 7fbc51e00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbc380a3c40 0x7fbc380a6100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0104e0 tx=0x7fbc3c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:38.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:38.928+0000 7fbc4aa00640 1 -- 172.21.15.94:0/2147407507 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbc4c016030 con 0x7fbc5410ab40 2024-04-26T20:27:39.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.174+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc5402dad0 con 0x7fbc5410ab40 2024-04-26T20:27:39.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.176+0000 7fbc4aa00640 1 -- 172.21.15.94:0/2147407507 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11878 (secure 0 0 0) 0x7fbc4c09c850 con 0x7fbc5410ab40 2024-04-26T20:27:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:27:39.177 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:12:34.670796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24562,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24541,24562]},"id":1}]} 2024-04-26T20:27:39.177 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 25 2024-04-26T20:27:39.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.178+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbc380a3c40 msgr2=0x7fbc380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.178+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbc380a3c40 0x7fbc380a6100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0104e0 tx=0x7fbc3c00b410 comp rx=0 tx=0).stop 2024-04-26T20:27:39.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.178+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc5410ab40 msgr2=0x7fbc541a6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.178+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc5410ab40 0x7fbc541a6700 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00e940 tx=0x7fbc4c00ee00 comp rx=0 tx=0).stop 2024-04-26T20:27:39.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.179+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 shutdown_connections 2024-04-26T20:27:39.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.179+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fbc380a3c40 0x7fbc380a6100 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.179+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbc5410ab40 0x7fbc541a6700 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.179+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc5410a180 0x7fbc5419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.179+0000 7fbc58e00640 1 --2- 172.21.15.94:0/2147407507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc54108f90 0x7fbc5419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.179+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 >> 172.21.15.94:0/2147407507 conn(0x7fbc54077f40 msgr2=0x7fbc5410d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:39.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.179+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 shutdown_connections 2024-04-26T20:27:39.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.179+0000 7fbc58e00640 1 -- 172.21.15.94:0/2147407507 wait complete. 2024-04-26T20:27:39.198 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:27:39.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.403+0000 7f9962800640 1 -- 172.21.15.94:0/3781834837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f995c106540 msgr2=0x7f995c10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.403+0000 7f9962800640 1 --2- 172.21.15.94:0/3781834837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f995c106540 0x7f995c10cdd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9954009ca0 tx=0x7f995402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.404+0000 7f9962800640 1 -- 172.21.15.94:0/3781834837 shutdown_connections 2024-04-26T20:27:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.404+0000 7f9962800640 1 --2- 172.21.15.94:0/3781834837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f995c106540 0x7f995c10cdd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.404+0000 7f9962800640 1 --2- 172.21.15.94:0/3781834837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f995c105b80 0x7f995c106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.404+0000 7f9962800640 1 --2- 172.21.15.94:0/3781834837 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f995c104990 0x7f995c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.404+0000 7f9962800640 1 -- 172.21.15.94:0/3781834837 >> 172.21.15.94:0/3781834837 conn(0x7f995c0fff00 msgr2=0x7f995c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.404+0000 7f9962800640 1 -- 172.21.15.94:0/3781834837 shutdown_connections 2024-04-26T20:27:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.404+0000 7f9962800640 1 -- 172.21.15.94:0/3781834837 wait complete. 2024-04-26T20:27:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9962800640 1 Processor -- start 2024-04-26T20:27:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9962800640 1 -- start start 2024-04-26T20:27:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9962800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f995c104990 0x7f995c19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9962800640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f995c105b80 0x7f995c19b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9962800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f995c106540 0x7f995c1a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9962800640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f995c110510 con 0x7f995c105b80 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9962800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f995c110390 con 0x7f995c106540 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9962800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f995c110690 con 0x7f995c104990 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9960a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f995c105b80 0x7f995c19b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9960a00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f995c105b80 0x7f995c19b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40562/0 (socket says 172.21.15.94:40562) 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9960a00640 1 -- 172.21.15.94:0/1025132849 learned_addr learned my addr 172.21.15.94:0/1025132849 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9961400640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f995c104990 0x7f995c19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.405+0000 7f9961e00640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f995c106540 0x7f995c1a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.406+0000 7f9960a00640 1 -- 172.21.15.94:0/1025132849 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f995c104990 msgr2=0x7f995c19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.406+0000 7f9960a00640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f995c104990 0x7f995c19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.406+0000 7f9960a00640 1 -- 172.21.15.94:0/1025132849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f995c106540 msgr2=0x7f995c1a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.406+0000 7f9960a00640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f995c106540 0x7f995c1a20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.406+0000 7f9960a00640 1 -- 172.21.15.94:0/1025132849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995c1a2740 con 0x7f995c105b80 2024-04-26T20:27:39.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.406+0000 7f9960a00640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f995c105b80 0x7f995c19b060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f994800e9f0 tx=0x7f994800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:39.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.406+0000 7f995a000640 1 -- 172.21.15.94:0/1025132849 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994800cd80 con 0x7f995c105b80 2024-04-26T20:27:39.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.406+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f995c1a2a30 con 0x7f995c105b80 2024-04-26T20:27:39.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.406+0000 7f995a000640 1 -- 172.21.15.94:0/1025132849 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99480054d0 con 0x7f995c105b80 2024-04-26T20:27:39.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.407+0000 7f995a000640 1 -- 172.21.15.94:0/1025132849 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9948018630 con 0x7f995c105b80 2024-04-26T20:27:39.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.407+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f995c1a2f30 con 0x7f995c105b80 2024-04-26T20:27:39.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.408+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f995c10bcb0 con 0x7f995c105b80 2024-04-26T20:27:39.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.410+0000 7f995a000640 1 -- 172.21.15.94:0/1025132849 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9948004e70 con 0x7f995c105b80 2024-04-26T20:27:39.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.411+0000 7f995a000640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f99440a3c40 0x7f99440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:39.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.411+0000 7f995a000640 1 -- 172.21.15.94:0/1025132849 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7f99480d0ee0 con 0x7f995c105b80 2024-04-26T20:27:39.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.411+0000 7f9961400640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f99440a3c40 0x7f99440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:39.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.412+0000 7f9961400640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f99440a3c40 0x7f99440a6100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f995c1059e0 tx=0x7f9950005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:39.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.415+0000 7f995a000640 1 -- 172.21.15.94:0/1025132849 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f994809d330 con 0x7f995c105b80 2024-04-26T20:27:39.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:39 smithi103 ceph-mon[32635]: pgmap v909: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:39.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:39 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2345012636' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:27:39.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:39 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2147407507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:27:39.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:39 smithi204 ceph-mon[32618]: pgmap v909: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:39.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:39 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2345012636' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:27:39.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:39 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2147407507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:27:39.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.663+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f995c02dd20 con 0x7f995c105b80 2024-04-26T20:27:39.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.664+0000 7f995a000640 1 -- 172.21.15.94:0/1025132849 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11878 (secure 0 0 0) 0x7f994809cc30 con 0x7f995c105b80 2024-04-26T20:27:39.666 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:27:39.666 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3305420604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":3305420604},{"type":"v1","addr":"172.21.15.94:6835","nonce":3305420604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6837/3734002100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6836","nonce":3734002100},{"type":"v1","addr":"172.21.15.94:6837","nonce":3734002100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6839/2249987305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6838","nonce":2249987305},{"type":"v1","addr":"172.21.15.94:6839","nonce":2249987305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3414557921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3414557921},{"type":"v1","addr":"172.21.15.204:6833","nonce":3414557921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/695386981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":695386981},{"type":"v1","addr":"172.21.15.103:6833","nonce":695386981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/4015617666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":4015617666},{"type":"v1","addr":"172.21.15.103:6835","nonce":4015617666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:08:18.399846+0000","modified":"2024-04-26T20:12:34.670796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24562,"mds_2":24541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6835/4163398350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4163398350},{"type":"v1","addr":"172.21.15.204:6835","nonce":4163398350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/820565196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":820565196},{"type":"v1","addr":"172.21.15.204:6837","nonce":820565196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/3697356689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":3697356689},{"type":"v1","addr":"172.21.15.103:6837","nonce":3697356689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6839/3633033106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3633033106},{"type":"v1","addr":"172.21.15.204:6839","nonce":3633033106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":22,"addr":"172.21.15.94:6841/989331046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6840","nonce":989331046},{"type":"v1","addr":"172.21.15.94:6841","nonce":989331046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6839/1048742250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1048742250},{"type":"v1","addr":"172.21.15.103:6839","nonce":1048742250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24541,24562]},"id":1}]} 2024-04-26T20:27:39.666 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 25 2024-04-26T20:27:39.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f99440a3c40 msgr2=0x7f99440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f99440a3c40 0x7f99440a6100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f995c1059e0 tx=0x7f9950005f90 comp rx=0 tx=0).stop 2024-04-26T20:27:39.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f995c105b80 msgr2=0x7f995c19b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f995c105b80 0x7f995c19b060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f994800e9f0 tx=0x7f994800edf0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 shutdown_connections 2024-04-26T20:27:39.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f99440a3c40 0x7f99440a6100 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f995c106540 0x7f995c1a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f995c105b80 0x7f995c19b060 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 --2- 172.21.15.94:0/1025132849 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f995c104990 0x7f995c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 >> 172.21.15.94:0/1025132849 conn(0x7f995c0fff00 msgr2=0x7f995c108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:39.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.667+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 shutdown_connections 2024-04-26T20:27:39.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.668+0000 7f9962800640 1 -- 172.21.15.94:0/1025132849 wait complete. 2024-04-26T20:27:39.684 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:39 smithi094 ceph-mon[28050]: pgmap v909: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-26T20:27:39.685 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:39 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2345012636' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:27:39.685 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:39 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2147407507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:27:39.688 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:27:39.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.882+0000 7ff9aee00640 1 -- 172.21.15.94:0/26731316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a8100c20 msgr2=0x7ff9a8101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.882+0000 7ff9aee00640 1 --2- 172.21.15.94:0/26731316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a8100c20 0x7ff9a8101040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff998009ca0 tx=0x7ff99802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.882+0000 7ff9aee00640 1 -- 172.21.15.94:0/26731316 shutdown_connections 2024-04-26T20:27:39.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.882+0000 7ff9aee00640 1 --2- 172.21.15.94:0/26731316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9a81045b0 0x7ff9a810ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.882+0000 7ff9aee00640 1 --2- 172.21.15.94:0/26731316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9a8103bf0 0x7ff9a8104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.882+0000 7ff9aee00640 1 --2- 172.21.15.94:0/26731316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a8100c20 0x7ff9a8101040 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.882+0000 7ff9aee00640 1 -- 172.21.15.94:0/26731316 >> 172.21.15.94:0/26731316 conn(0x7ff9a80fc550 msgr2=0x7ff9a80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:39.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.882+0000 7ff9aee00640 1 -- 172.21.15.94:0/26731316 shutdown_connections 2024-04-26T20:27:39.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.882+0000 7ff9aee00640 1 -- 172.21.15.94:0/26731316 wait complete. 2024-04-26T20:27:39.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.883+0000 7ff9aee00640 1 Processor -- start 2024-04-26T20:27:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9aee00640 1 -- start start 2024-04-26T20:27:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9aee00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9a8100c20 0x7ff9a8198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9aee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9a8103bf0 0x7ff9a81990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9aee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a81045b0 0x7ff9a81a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9aee00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a810e4a0 con 0x7ff9a8100c20 2024-04-26T20:27:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9aee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a810e320 con 0x7ff9a81045b0 2024-04-26T20:27:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9aee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a810e620 con 0x7ff9a8103bf0 2024-04-26T20:27:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9a7e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9a8100c20 0x7ff9a8198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9a7e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9a8100c20 0x7ff9a8198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40588/0 (socket says 172.21.15.94:40588) 2024-04-26T20:27:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9a7e00640 1 -- 172.21.15.94:0/1896812028 learned_addr learned my addr 172.21.15.94:0/1896812028 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff9ace00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a81045b0 0x7ff9a81a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.884+0000 7ff99f400640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9a8103bf0 0x7ff9a81990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9a7e00640 1 -- 172.21.15.94:0/1896812028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9a8103bf0 msgr2=0x7ff9a81990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9a7e00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9a8103bf0 0x7ff9a81990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9a7e00640 1 -- 172.21.15.94:0/1896812028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a81045b0 msgr2=0x7ff9a81a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9a7e00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a81045b0 0x7ff9a81a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9a7e00640 1 -- 172.21.15.94:0/1896812028 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a81a07a0 con 0x7ff9a8100c20 2024-04-26T20:27:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9ace00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a81045b0 0x7ff9a81a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:27:39.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9a7e00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9a8100c20 0x7ff9a8198b80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff998004af0 tx=0x7ff998003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:39.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9a5600640 1 -- 172.21.15.94:0/1896812028 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998004240 con 0x7ff9a8100c20 2024-04-26T20:27:39.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a81a0a30 con 0x7ff9a8100c20 2024-04-26T20:27:39.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9a5600640 1 -- 172.21.15.94:0/1896812028 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff998003e40 con 0x7ff9a8100c20 2024-04-26T20:27:39.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.885+0000 7ff9a5600640 1 -- 172.21.15.94:0/1896812028 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99802fbc0 con 0x7ff9a8100c20 2024-04-26T20:27:39.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.886+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a81a0fd0 con 0x7ff9a8100c20 2024-04-26T20:27:39.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.887+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a8109d90 con 0x7ff9a8100c20 2024-04-26T20:27:39.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.888+0000 7ff9a5600640 1 -- 172.21.15.94:0/1896812028 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff99802fd60 con 0x7ff9a8100c20 2024-04-26T20:27:39.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.890+0000 7ff9a5600640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff98c0a3cc0 0x7ff98c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:39.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.890+0000 7ff9a5600640 1 -- 172.21.15.94:0/1896812028 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7ff9980f1c90 con 0x7ff9a8100c20 2024-04-26T20:27:39.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.890+0000 7ff99f400640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff98c0a3cc0 0x7ff98c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:39.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.891+0000 7ff99f400640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff98c0a3cc0 0x7ff98c0a6180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff9a819a090 tx=0x7ff990005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:39.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:39.894+0000 7ff9a5600640 1 -- 172.21.15.94:0/1896812028 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff9980be1f0 con 0x7ff9a8100c20 2024-04-26T20:27:40.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.089+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9a802dd20 con 0x7ff9a8100c20 2024-04-26T20:27:40.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.092+0000 7ff9a5600640 1 -- 172.21.15.94:0/1896812028 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24879 (secure 0 0 0) 0x7ff9980bdaf0 con 0x7ff9a8100c20 2024-04-26T20:27:40.094 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-04-26T20:27:40.094 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"420d0bd6-0407-11ef-bc93-c7b262605968","created":"2024-04-26T19:58:57.906790+0000","modified":"2024-04-26T20:10:13.755106+0000","last_up_change":"2024-04-26T20:07:10.291287+0000","last_in_change":"2024-04-26T20:06:49.842701+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:02:53.592939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:08:13.904332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:08:15.670247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:08:23.682553+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6d79ff6-dda1-4462-b2e6-eac0c688fef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6803","nonce":3381923901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6805","nonce":3381923901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6809","nonce":3381923901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":3381923901},{"type":"v1","addr":"172.21.15.94:6807","nonce":3381923901}]},"public_addr":"172.21.15.94:6803/3381923901","cluster_addr":"172.21.15.94:6805/3381923901","heartbeat_back_addr":"172.21.15.94:6809/3381923901","heartbeat_front_addr":"172.21.15.94:6807/3381923901","state":["exists","up"]},{"osd":1,"uuid":"eef6efa3-ce90-4d2d-8fb8-4a2665258434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6801","nonce":1497163956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6803","nonce":1497163956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6807","nonce":1497163956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1497163956},{"type":"v1","addr":"172.21.15.103:6805","nonce":1497163956}]},"public_addr":"172.21.15.103:6801/1497163956","cluster_addr":"172.21.15.103:6803/1497163956","heartbeat_back_addr":"172.21.15.103:6807/1497163956","heartbeat_front_addr":"172.21.15.103:6805/1497163956","state":["exists","up"]},{"osd":2,"uuid":"939eb12b-359a-4c5e-8e2f-c94e6512050e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6801","nonce":1401040530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6803","nonce":1401040530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6807","nonce":1401040530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":1401040530},{"type":"v1","addr":"172.21.15.204:6805","nonce":1401040530}]},"public_addr":"172.21.15.204:6801/1401040530","cluster_addr":"172.21.15.204:6803/1401040530","heartbeat_back_addr":"172.21.15.204:6807/1401040530","heartbeat_front_addr":"172.21.15.204:6805/1401040530","state":["exists","up"]},{"osd":3,"uuid":"aa4d0799-6fb8-439e-a987-91a977ac9172","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6811","nonce":2184382995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6813","nonce":2184382995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6817","nonce":2184382995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":2184382995},{"type":"v1","addr":"172.21.15.94:6815","nonce":2184382995}]},"public_addr":"172.21.15.94:6811/2184382995","cluster_addr":"172.21.15.94:6813/2184382995","heartbeat_back_addr":"172.21.15.94:6817/2184382995","heartbeat_front_addr":"172.21.15.94:6815/2184382995","state":["exists","up"]},{"osd":4,"uuid":"b5d3851f-30c6-4ec7-a8da-0b949c46c6c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6809","nonce":3296783557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6811","nonce":3296783557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6815","nonce":3296783557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3296783557},{"type":"v1","addr":"172.21.15.103:6813","nonce":3296783557}]},"public_addr":"172.21.15.103:6809/3296783557","cluster_addr":"172.21.15.103:6811/3296783557","heartbeat_back_addr":"172.21.15.103:6815/3296783557","heartbeat_front_addr":"172.21.15.103:6813/3296783557","state":["exists","up"]},{"osd":5,"uuid":"c3e5aa68-5a1c-449c-b613-e00d934eb78e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6809","nonce":1603790096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6811","nonce":1603790096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6815","nonce":1603790096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1603790096},{"type":"v1","addr":"172.21.15.204:6813","nonce":1603790096}]},"public_addr":"172.21.15.204:6809/1603790096","cluster_addr":"172.21.15.204:6811/1603790096","heartbeat_back_addr":"172.21.15.204:6815/1603790096","heartbeat_front_addr":"172.21.15.204:6813/1603790096","state":["exists","up"]},{"osd":6,"uuid":"438ff921-8607-404a-bffd-7234940076bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6819","nonce":4225405603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6821","nonce":4225405603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6825","nonce":4225405603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4225405603},{"type":"v1","addr":"172.21.15.94:6823","nonce":4225405603}]},"public_addr":"172.21.15.94:6819/4225405603","cluster_addr":"172.21.15.94:6821/4225405603","heartbeat_back_addr":"172.21.15.94:6825/4225405603","heartbeat_front_addr":"172.21.15.94:6823/4225405603","state":["exists","up"]},{"osd":7,"uuid":"e1df6748-fae6-4f37-806d-a6093ec9ca60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6817","nonce":1379914194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6819","nonce":1379914194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6823","nonce":1379914194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1379914194},{"type":"v1","addr":"172.21.15.103:6821","nonce":1379914194}]},"public_addr":"172.21.15.103:6817/1379914194","cluster_addr":"172.21.15.103:6819/1379914194","heartbeat_back_addr":"172.21.15.103:6823/1379914194","heartbeat_front_addr":"172.21.15.103:6821/1379914194","state":["exists","up"]},{"osd":8,"uuid":"733fdc3e-cdc8-4f38-afd9-6bd2012145af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6817","nonce":1919172767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6819","nonce":1919172767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6823","nonce":1919172767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1919172767},{"type":"v1","addr":"172.21.15.204:6821","nonce":1919172767}]},"public_addr":"172.21.15.204:6817/1919172767","cluster_addr":"172.21.15.204:6819/1919172767","heartbeat_back_addr":"172.21.15.204:6823/1919172767","heartbeat_front_addr":"172.21.15.204:6821/1919172767","state":["exists","up"]},{"osd":9,"uuid":"6bdd2b36-7b98-470b-b0c1-659406d966b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6827","nonce":4013917902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6829","nonce":4013917902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6833","nonce":4013917902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":4013917902},{"type":"v1","addr":"172.21.15.94:6831","nonce":4013917902}]},"public_addr":"172.21.15.94:6827/4013917902","cluster_addr":"172.21.15.94:6829/4013917902","heartbeat_back_addr":"172.21.15.94:6833/4013917902","heartbeat_front_addr":"172.21.15.94:6831/4013917902","state":["exists","up"]},{"osd":10,"uuid":"95a7b28b-3f5d-4fce-a7eb-b4c90eb162a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6825","nonce":2183890761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6827","nonce":2183890761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6831","nonce":2183890761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2183890761},{"type":"v1","addr":"172.21.15.103:6829","nonce":2183890761}]},"public_addr":"172.21.15.103:6825/2183890761","cluster_addr":"172.21.15.103:6827/2183890761","heartbeat_back_addr":"172.21.15.103:6831/2183890761","heartbeat_front_addr":"172.21.15.103:6829/2183890761","state":["exists","up"]},{"osd":11,"uuid":"dc362076-e2fb-4994-97c6-08e2cee8e0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6825","nonce":1860743299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6827","nonce":1860743299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6831","nonce":1860743299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":1860743299},{"type":"v1","addr":"172.21.15.204:6829","nonce":1860743299}]},"public_addr":"172.21.15.204:6825/1860743299","cluster_addr":"172.21.15.204:6827/1860743299","heartbeat_back_addr":"172.21.15.204:6831/1860743299","heartbeat_front_addr":"172.21.15.204:6829/1860743299","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:02.988713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:27.818414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:02:51.185282+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:24.196018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:03:50.198064+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:14.827108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:04:49.168874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:15.704132+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:05:40.627261+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:14.801227+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:06:42.016111+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:07:07.541090+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:0/3877259400":"2024-04-27T20:00:11.413271+0000","172.21.15.94:0/3361956945":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3651438916":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6801/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1688404742":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/2041870658":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/442392134":"2024-04-27T19:59:38.717857+0000","172.21.15.94:0/1853030572":"2024-04-27T20:00:11.413271+0000","172.21.15.94:6800/3005317891":"2024-04-27T19:59:38.717857+0000","172.21.15.94:6800/3651438916":"2024-04-27T20:00:11.413271+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:27:40.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff98c0a3cc0 msgr2=0x7ff98c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff98c0a3cc0 0x7ff98c0a6180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff9a819a090 tx=0x7ff990005f90 comp rx=0 tx=0).stop 2024-04-26T20:27:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9a8100c20 msgr2=0x7ff9a8198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9a8100c20 0x7ff9a8198b80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff998004af0 tx=0x7ff998003c20 comp rx=0 tx=0).stop 2024-04-26T20:27:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 shutdown_connections 2024-04-26T20:27:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7ff98c0a3cc0 0x7ff98c0a6180 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a81045b0 0x7ff9a81a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9a8103bf0 0x7ff9a81990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 --2- 172.21.15.94:0/1896812028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9a8100c20 0x7ff9a8198b80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 >> 172.21.15.94:0/1896812028 conn(0x7ff9a80fc550 msgr2=0x7ff9a8106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:40.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 shutdown_connections 2024-04-26T20:27:40.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.095+0000 7ff9aee00640 1 -- 172.21.15.94:0/1896812028 wait complete. 2024-04-26T20:27:40.117 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T20:27:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.303+0000 7fc351000640 1 -- 172.21.15.94:0/1969252498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 msgr2=0x7fc34c0757d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.303+0000 7fc351000640 1 --2- 172.21.15.94:0/1969252498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 0x7fc34c0757d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc334009860 tx=0x7fc33402a340 comp rx=0 tx=0).stop 2024-04-26T20:27:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.303+0000 7fc351000640 1 -- 172.21.15.94:0/1969252498 shutdown_connections 2024-04-26T20:27:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.303+0000 7fc351000640 1 --2- 172.21.15.94:0/1969252498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc34c10a620 0x7fc34c10ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.303+0000 7fc351000640 1 --2- 172.21.15.94:0/1969252498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 0x7fc34c0757d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.303+0000 7fc351000640 1 --2- 172.21.15.94:0/1969252498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc34c076930 0x7fc34c074d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.304+0000 7fc351000640 1 -- 172.21.15.94:0/1969252498 >> 172.21.15.94:0/1969252498 conn(0x7fc34c0fdcc0 msgr2=0x7fc34c100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.304+0000 7fc351000640 1 -- 172.21.15.94:0/1969252498 shutdown_connections 2024-04-26T20:27:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.304+0000 7fc351000640 1 -- 172.21.15.94:0/1969252498 wait complete. 2024-04-26T20:27:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc351000640 1 Processor -- start 2024-04-26T20:27:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc351000640 1 -- start start 2024-04-26T20:27:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc351000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 0x7fc34c19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc351000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc34c076930 0x7fc34c19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc351000640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc34c10a620 0x7fc34c1a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc351000640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7fc34c110480 con 0x7fc34c10a620 2024-04-26T20:27:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc351000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc34c110300 con 0x7fc34c075350 2024-04-26T20:27:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc351000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc34c110600 con 0x7fc34c076930 2024-04-26T20:27:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc34b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 0x7fc34c19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc34be00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc34c10a620 0x7fc34c1a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc34b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 0x7fc34c19ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:33844/0 (socket says 172.21.15.94:33844) 2024-04-26T20:27:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc34b400640 1 -- 172.21.15.94:0/4129622828 learned_addr learned my addr 172.21.15.94:0/4129622828 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.305+0000 7fc34aa00640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc34c076930 0x7fc34c19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.306+0000 7fc34b400640 1 -- 172.21.15.94:0/4129622828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc34c076930 msgr2=0x7fc34c19b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.306+0000 7fc34b400640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc34c076930 0x7fc34c19b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.306+0000 7fc34b400640 1 -- 172.21.15.94:0/4129622828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc34c10a620 msgr2=0x7fc34c1a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.306+0000 7fc34b400640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc34c10a620 0x7fc34c1a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.306+0000 7fc34b400640 1 -- 172.21.15.94:0/4129622828 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34c1a27a0 con 0x7fc34c075350 2024-04-26T20:27:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.306+0000 7fc34b400640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 0x7fc34c19ab80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc33c00efb0 tx=0x7fc33c00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.306+0000 7fc343e00640 1 -- 172.21.15.94:0/4129622828 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c01b070 con 0x7fc34c075350 2024-04-26T20:27:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.306+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc34c1a2a90 con 0x7fc34c075350 2024-04-26T20:27:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.307+0000 7fc343e00640 1 -- 172.21.15.94:0/4129622828 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc33c00ba20 con 0x7fc34c075350 2024-04-26T20:27:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.307+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc34c1a3000 con 0x7fc34c075350 2024-04-26T20:27:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.307+0000 7fc343e00640 1 -- 172.21.15.94:0/4129622828 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c016b00 con 0x7fc34c075350 2024-04-26T20:27:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.308+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3280052c0 con 0x7fc34c075350 2024-04-26T20:27:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.309+0000 7fc343e00640 1 -- 172.21.15.94:0/4129622828 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc33c01e070 con 0x7fc34c075350 2024-04-26T20:27:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.311+0000 7fc343e00640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc3300a3cc0 0x7fc3300a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.311+0000 7fc343e00640 1 -- 172.21.15.94:0/4129622828 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9208+0+0 (secure 0 0 0) 0x7fc33c0d08c0 con 0x7fc34c075350 2024-04-26T20:27:40.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.311+0000 7fc34aa00640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc3300a3cc0 0x7fc3300a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:40.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.312+0000 7fc34aa00640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc3300a3cc0 0x7fc3300a6180 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc334000c00 tx=0x7fc334030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:40.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.313+0000 7fc343e00640 1 -- 172.21.15.94:0/4129622828 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc33c09c6c0 con 0x7fc34c075350 2024-04-26T20:27:40.482 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:40 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1025132849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:27:40.482 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:40 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/1896812028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:27:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.557+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc328005880 con 0x7fc34c075350 2024-04-26T20:27:40.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.567+0000 7fc343e00640 1 -- 172.21.15.94:0/4129622828 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7fc33c0a09c0 con 0x7fc34c075350 2024-04-26T20:27:40.568 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T20:27:40.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.571+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc3300a3cc0 msgr2=0x7fc3300a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.571+0000 7fc351000640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc3300a3cc0 0x7fc3300a6180 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc334000c00 tx=0x7fc334030470 comp rx=0 tx=0).stop 2024-04-26T20:27:40.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.571+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 msgr2=0x7fc34c19ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.571+0000 7fc351000640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 0x7fc34c19ab80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc33c00efb0 tx=0x7fc33c00c800 comp rx=0 tx=0).stop 2024-04-26T20:27:40.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.572+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 shutdown_connections 2024-04-26T20:27:40.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.572+0000 7fc351000640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7fc3300a3cc0 0x7fc3300a6180 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.572+0000 7fc351000640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc34c10a620 0x7fc34c1a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.572+0000 7fc351000640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc34c076930 0x7fc34c19b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.572+0000 7fc351000640 1 --2- 172.21.15.94:0/4129622828 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc34c075350 0x7fc34c19ab80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.572+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 >> 172.21.15.94:0/4129622828 conn(0x7fc34c0fdcc0 msgr2=0x7fc34c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:40.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.572+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 shutdown_connections 2024-04-26T20:27:40.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.572+0000 7fc351000640 1 -- 172.21.15.94:0/4129622828 wait complete. 2024-04-26T20:27:40.598 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T20:27:40.606 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1025132849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:27:40.607 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/1896812028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:27:40.607 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1025132849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:27:40.607 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/1896812028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:27:40.737 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:27:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a[66827]: ignoring --setuser ceph since I am not root 2024-04-26T20:27:40.737 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:27:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a[66827]: ignoring --setgroup ceph since I am not root 2024-04-26T20:27:40.737 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:27:40 smithi094 ceph-mds[66832]: -- 172.21.15.94:0/3712982412 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558805399a40 con 0x55880536c800 2024-04-26T20:27:40.737 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:27:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a[66827]: starting mds.a at 2024-04-26T20:27:40.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a[28046]: 2024-04-26T20:27:40.559+0000 7f5f58a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T20:27:40.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.772+0000 7f1d81400640 1 -- 172.21.15.94:0/3901468943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d7c103a30 msgr2=0x7f1d7c103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.772+0000 7f1d81400640 1 --2- 172.21.15.94:0/3901468943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d7c103a30 0x7f1d7c103eb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009900 tx=0x7f1d6402a400 comp rx=0 tx=0).stop 2024-04-26T20:27:40.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.772+0000 7f1d81400640 1 -- 172.21.15.94:0/3901468943 shutdown_connections 2024-04-26T20:27:40.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.772+0000 7f1d81400640 1 --2- 172.21.15.94:0/3901468943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c1043f0 0x7f1d7c10ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.772+0000 7f1d81400640 1 --2- 172.21.15.94:0/3901468943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d7c103a30 0x7f1d7c103eb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.772+0000 7f1d81400640 1 --2- 172.21.15.94:0/3901468943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d7c102840 0x7f1d7c102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.772+0000 7f1d81400640 1 -- 172.21.15.94:0/3901468943 >> 172.21.15.94:0/3901468943 conn(0x7f1d7c0fddd0 msgr2=0x7f1d7c1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:40.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.772+0000 7f1d81400640 1 -- 172.21.15.94:0/3901468943 shutdown_connections 2024-04-26T20:27:40.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.772+0000 7f1d81400640 1 -- 172.21.15.94:0/3901468943 wait complete. 2024-04-26T20:27:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.773+0000 7f1d81400640 1 Processor -- start 2024-04-26T20:27:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.773+0000 7f1d81400640 1 -- start start 2024-04-26T20:27:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d81400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d7c102840 0x7f1d7c19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d81400640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d7c103a30 0x7f1d7c19b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d81400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c1043f0 0x7f1d7c1a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d81400640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c10e370 con 0x7f1d7c103a30 2024-04-26T20:27:40.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d81400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c10e1f0 con 0x7f1d7c1043f0 2024-04-26T20:27:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d81400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c10e4f0 con 0x7f1d7c102840 2024-04-26T20:27:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d73e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d7c103a30 0x7f1d7c19b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d80a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c1043f0 0x7f1d7c1a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d7be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d7c102840 0x7f1d7c19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d80a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c1043f0 0x7f1d7c1a2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:33868/0 (socket says 172.21.15.94:33868) 2024-04-26T20:27:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d73e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d7c103a30 0x7f1d7c19b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40654/0 (socket says 172.21.15.94:40654) 2024-04-26T20:27:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.774+0000 7f1d80a00640 1 -- 172.21.15.94:0/3526526478 learned_addr learned my addr 172.21.15.94:0/3526526478 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:40.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d7be00640 1 -- 172.21.15.94:0/3526526478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c1043f0 msgr2=0x7f1d7c1a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d7be00640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c1043f0 0x7f1d7c1a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d7be00640 1 -- 172.21.15.94:0/3526526478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d7c103a30 msgr2=0x7f1d7c19b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d7be00640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d7c103a30 0x7f1d7c19b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d7be00640 1 -- 172.21.15.94:0/3526526478 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7c1a28c0 con 0x7f1d7c102840 2024-04-26T20:27:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d7be00640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d7c102840 0x7f1d7c19ac80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00e9c0 tx=0x7f1d6c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d79600640 1 -- 172.21.15.94:0/3526526478 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00ccc0 con 0x7f1d7c102840 2024-04-26T20:27:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7c1a2bb0 con 0x7f1d7c102840 2024-04-26T20:27:40.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d79600640 1 -- 172.21.15.94:0/3526526478 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d6c0035f0 con 0x7f1d7c102840 2024-04-26T20:27:40.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d79600640 1 -- 172.21.15.94:0/3526526478 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00ce60 con 0x7f1d7c102840 2024-04-26T20:27:40.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.775+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7c1a30d0 con 0x7f1d7c102840 2024-04-26T20:27:40.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.776+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d7c1a32a0 con 0x7f1d7c102840 2024-04-26T20:27:40.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.780+0000 7f1d79600640 1 -- 172.21.15.94:0/3526526478 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1d6c004000 con 0x7f1d7c102840 2024-04-26T20:27:40.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.781+0000 7f1d79600640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1d600a3c40 0x7f1d600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:40.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.782+0000 7f1d79600640 1 -- 172.21.15.94:0/3526526478 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f1d6c0d0ec0 con 0x7f1d7c102840 2024-04-26T20:27:40.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.782+0000 7f1d73e00640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1d600a3c40 0x7f1d600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:40.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.782+0000 7f1d73e00640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1d600a3c40 0x7f1d600a6100 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1d64000c00 tx=0x7f1d640314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:40.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:40.786+0000 7f1d79600640 1 -- 172.21.15.94:0/3526526478 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1d6c09d210 con 0x7f1d7c102840 2024-04-26T20:27:40.882 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c[63186]: ignoring --setuser ceph since I am not root 2024-04-26T20:27:40.882 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c[63186]: ignoring --setgroup ceph since I am not root 2024-04-26T20:27:40.882 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-mds[63190]: -- 172.21.15.204:0/4034440538 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610b218da40 con 0x5610b2160c00 2024-04-26T20:27:40.882 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c[63186]: starting mds.c at 2024-04-26T20:27:40.883 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f[62880]: ignoring --setuser ceph since I am not root 2024-04-26T20:27:40.883 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f[62880]: ignoring --setgroup ceph since I am not root 2024-04-26T20:27:40.883 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-mds[62884]: -- 172.21.15.204:0/1394889076 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55864917ba40 con 0x55864914e800 2024-04-26T20:27:40.883 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f[62880]: starting mds.f at 2024-04-26T20:27:40.883 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i[63496]: ignoring --setuser ceph since I am not root 2024-04-26T20:27:40.883 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i[63496]: ignoring --setgroup ceph since I am not root 2024-04-26T20:27:40.884 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-mds[63500]: -- 172.21.15.204:0/1635553112 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555e4e005a40 con 0x555e4dfd8800 2024-04-26T20:27:40.884 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:27:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i[63496]: starting mds.i at 2024-04-26T20:27:41.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.022+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1d7c1a3470 con 0x7f1d7c102840 2024-04-26T20:27:41.034 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b[63233]: ignoring --setuser ceph since I am not root 2024-04-26T20:27:41.034 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b[63233]: ignoring --setgroup ceph since I am not root 2024-04-26T20:27:41.034 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-mds[63237]: -- 172.21.15.103:0/3448128699 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558b78571a40 con 0x558b78545000 2024-04-26T20:27:41.034 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b[63233]: starting mds.b at 2024-04-26T20:27:41.035 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b[63233]: 2024-04-26T20:27:40.610+0000 7f2e0521e700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6838/0: (98) Address already in use 2024-04-26T20:27:41.035 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h[62919]: ignoring --setuser ceph since I am not root 2024-04-26T20:27:41.035 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h[62919]: ignoring --setgroup ceph since I am not root 2024-04-26T20:27:41.035 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-mds[62923]: -- 172.21.15.103:0/2587484307 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d1d835a40 con 0x555d1d808c00 2024-04-26T20:27:41.035 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:27:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h[62919]: starting mds.h at 2024-04-26T20:27:41.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.569+0000 7f1d79600640 1 -- 172.21.15.94:0/3526526478 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7f1d6c09cb10 con 0x7f1d7c102840 2024-04-26T20:27:41.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.572+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1d600a3c40 msgr2=0x7f1d600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:41.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.572+0000 7f1d81400640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1d600a3c40 0x7f1d600a6100 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1d64000c00 tx=0x7f1d640314a0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.572+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d7c102840 msgr2=0x7f1d7c19ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:41.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.572+0000 7f1d81400640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d7c102840 0x7f1d7c19ac80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00e9c0 tx=0x7f1d6c00ee80 comp rx=0 tx=0).stop 2024-04-26T20:27:41.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.573+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 shutdown_connections 2024-04-26T20:27:41.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.573+0000 7f1d81400640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f1d600a3c40 0x7f1d600a6100 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.573+0000 7f1d81400640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c1043f0 0x7f1d7c1a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.573+0000 7f1d81400640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d7c103a30 0x7f1d7c19b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.573+0000 7f1d81400640 1 --2- 172.21.15.94:0/3526526478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d7c102840 0x7f1d7c19ac80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.573+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 >> 172.21.15.94:0/3526526478 conn(0x7f1d7c0fddd0 msgr2=0x7f1d7c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:41.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.574+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 shutdown_connections 2024-04-26T20:27:41.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.574+0000 7f1d81400640 1 -- 172.21.15.94:0/3526526478 wait complete. 2024-04-26T20:27:41.597 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: pgmap v910: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-26T20:27:41.598 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/4129622828' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:27:41.598 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:27:41.598 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T20:27:41.598 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T20:27:41.598 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T20:27:41.598 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T20:27:41.599 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T20:27:41.599 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3526526478' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:41.599 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:41 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:41.600 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T20:27:41.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: pgmap v910: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-26T20:27:41.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/4129622828' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:27:41.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:27:41.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T20:27:41.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T20:27:41.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T20:27:41.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T20:27:41.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T20:27:41.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3526526478' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:41.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:41 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: pgmap v910: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-26T20:27:41.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/4129622828' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:27:41.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:27:41.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T20:27:41.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T20:27:41.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T20:27:41.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T20:27:41.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T20:27:41.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3526526478' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:41.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:41 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.786+0000 7f1933200640 1 -- 172.21.15.94:0/1941174679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934109fc0 msgr2=0x7f193410a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.786+0000 7f1933200640 1 --2- 172.21.15.94:0/1941174679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934109fc0 0x7f193410a440 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1920009900 tx=0x7f192002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.786+0000 7f1933200640 1 -- 172.21.15.94:0/1941174679 shutdown_connections 2024-04-26T20:27:41.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.786+0000 7f1933200640 1 --2- 172.21.15.94:0/1941174679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f193410a980 0x7f19341112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.786+0000 7f1933200640 1 --2- 172.21.15.94:0/1941174679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934109fc0 0x7f193410a440 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.786+0000 7f1933200640 1 --2- 172.21.15.94:0/1941174679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1934108e70 0x7f1934109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.786+0000 7f1933200640 1 -- 172.21.15.94:0/1941174679 >> 172.21.15.94:0/1941174679 conn(0x7f1934077e00 msgr2=0x7f193407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:41.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.786+0000 7f1933200640 1 -- 172.21.15.94:0/1941174679 shutdown_connections 2024-04-26T20:27:41.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.786+0000 7f1933200640 1 -- 172.21.15.94:0/1941174679 wait complete. 2024-04-26T20:27:41.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.787+0000 7f1933200640 1 Processor -- start 2024-04-26T20:27:41.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.787+0000 7f1933200640 1 -- start start 2024-04-26T20:27:41.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.787+0000 7f1933200640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934108e70 0x7f193419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:41.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.787+0000 7f1933200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1934109fc0 0x7f193419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:41.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.787+0000 7f1933200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f193410a980 0x7f19341a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:41.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.787+0000 7f1933200640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f19341149f0 con 0x7f1934108e70 2024-04-26T20:27:41.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.787+0000 7f1933200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1934114870 con 0x7f1934109fc0 2024-04-26T20:27:41.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.787+0000 7f1933200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1934114b70 con 0x7f193410a980 2024-04-26T20:27:41.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934108e70 0x7f193419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:41.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934108e70 0x7f193419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60392/0 (socket says 172.21.15.94:60392) 2024-04-26T20:27:41.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931e00640 1 -- 172.21.15.94:0/2018132148 learned_addr learned my addr 172.21.15.94:0/2018132148 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:41.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931400640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1934109fc0 0x7f193419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:41.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1932800640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f193410a980 0x7f19341a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:41.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931e00640 1 -- 172.21.15.94:0/2018132148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f193410a980 msgr2=0x7f19341a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:41.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931e00640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f193410a980 0x7f19341a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931e00640 1 -- 172.21.15.94:0/2018132148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1934109fc0 msgr2=0x7f193419f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:41.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931e00640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1934109fc0 0x7f193419f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:41.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931e00640 1 -- 172.21.15.94:0/2018132148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19341a6c80 con 0x7f1934108e70 2024-04-26T20:27:41.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.788+0000 7f1931e00640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934108e70 0x7f193419f060 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f192400e9f0 tx=0x7f192400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:41.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.789+0000 7f192aa00640 1 -- 172.21.15.94:0/2018132148 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192400cd80 con 0x7f1934108e70 2024-04-26T20:27:41.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.789+0000 7f192aa00640 1 -- 172.21.15.94:0/2018132148 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19240108e0 con 0x7f1934108e70 2024-04-26T20:27:41.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.789+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19341a6f70 con 0x7f1934108e70 2024-04-26T20:27:41.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.789+0000 7f192aa00640 1 -- 172.21.15.94:0/2018132148 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1924018980 con 0x7f1934108e70 2024-04-26T20:27:41.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.789+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19341a73e0 con 0x7f1934108e70 2024-04-26T20:27:41.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.790+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1934110180 con 0x7f1934108e70 2024-04-26T20:27:41.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.792+0000 7f192aa00640 1 -- 172.21.15.94:0/2018132148 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1924010430 con 0x7f1934108e70 2024-04-26T20:27:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.793+0000 7f192aa00640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f19180a3cc0 0x7f19180a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.794+0000 7f192aa00640 1 -- 172.21.15.94:0/2018132148 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9466+0+0 (secure 0 0 0) 0x7f19240d0dd0 con 0x7f1934108e70 2024-04-26T20:27:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.794+0000 7f1931400640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f19180a3cc0 0x7f19180a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:41.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.794+0000 7f1931400640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f19180a3cc0 0x7f19180a6180 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1920005ab0 tx=0x7f1920005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:41.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:41.798+0000 7f192aa00640 1 -- 172.21.15.94:0/2018132148 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f192409d120 con 0x7f1934108e70 2024-04-26T20:27:42.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.000+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f193402dad0 con 0x7f1934108e70 2024-04-26T20:27:42.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.577+0000 7f192aa00640 1 -- 172.21.15.94:0/2018132148 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) ==== 181+0+0 (secure 0 0 0) 0x7f192409ca20 con 0x7f1934108e70 2024-04-26T20:27:42.579 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-04-26T20:27:42.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.584+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f19180a3cc0 msgr2=0x7f19180a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:42.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.584+0000 7f1933200640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f19180a3cc0 0x7f19180a6180 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1920005ab0 tx=0x7f1920005a20 comp rx=0 tx=0).stop 2024-04-26T20:27:42.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.584+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934108e70 msgr2=0x7f193419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:42.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.584+0000 7f1933200640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934108e70 0x7f193419f060 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f192400e9f0 tx=0x7f192400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.585+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 shutdown_connections 2024-04-26T20:27:42.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.585+0000 7f1933200640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f19180a3cc0 0x7f19180a6180 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.585+0000 7f1933200640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f193410a980 0x7f19341a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.585+0000 7f1933200640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1934109fc0 0x7f193419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.585+0000 7f1933200640 1 --2- 172.21.15.94:0/2018132148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1934108e70 0x7f193419f060 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.585+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 >> 172.21.15.94:0/2018132148 conn(0x7f1934077e00 msgr2=0x7f193410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:42.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.586+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 shutdown_connections 2024-04-26T20:27:42.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.586+0000 7f1933200640 1 -- 172.21.15.94:0/2018132148 wait complete. 2024-04-26T20:27:42.605 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T20:27:42.605 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T20:27:42.605 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: Cluster is now healthy 2024-04-26T20:27:42.605 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:27:42.605 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.103:6837/2477484723,v1:172.21.15.103:6838/2477484723] up:boot 2024-04-26T20:27:42.605 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.204:6834/3492275136,v1:172.21.15.204:6835/3492275136] up:boot 2024-04-26T20:27:42.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.103:6836/1144594115,v1:172.21.15.103:6839/1144594115] up:boot 2024-04-26T20:27:42.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.204:6836/1832012233,v1:172.21.15.204:6838/1832012233] up:boot 2024-04-26T20:27:42.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.204:6837/1175426508,v1:172.21.15.204:6839/1175426508] up:boot 2024-04-26T20:27:42.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: mds.? [v2:172.21.15.94:6840/3605429688,v1:172.21.15.94:6841/3605429688] up:boot 2024-04-26T20:27:42.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: fsmap 12 up:standby 2024-04-26T20:27:42.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:27:42.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:27:42.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:27:42.606 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:27:42.607 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:27:42.607 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T20:27:42.607 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:42 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2018132148' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:42.609 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T20:27:42.631 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T20:27:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T20:27:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: Cluster is now healthy 2024-04-26T20:27:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:27:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.103:6837/2477484723,v1:172.21.15.103:6838/2477484723] up:boot 2024-04-26T20:27:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.204:6834/3492275136,v1:172.21.15.204:6835/3492275136] up:boot 2024-04-26T20:27:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.103:6836/1144594115,v1:172.21.15.103:6839/1144594115] up:boot 2024-04-26T20:27:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.204:6836/1832012233,v1:172.21.15.204:6838/1832012233] up:boot 2024-04-26T20:27:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.204:6837/1175426508,v1:172.21.15.204:6839/1175426508] up:boot 2024-04-26T20:27:42.632 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: mds.? [v2:172.21.15.94:6840/3605429688,v1:172.21.15.94:6841/3605429688] up:boot 2024-04-26T20:27:42.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: fsmap 12 up:standby 2024-04-26T20:27:42.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:27:42.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:27:42.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:27:42.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:27:42.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:27:42.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T20:27:42.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:42 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2018132148' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:42.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T20:27:42.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T20:27:42.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: Cluster is now healthy 2024-04-26T20:27:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:27:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.103:6837/2477484723,v1:172.21.15.103:6838/2477484723] up:boot 2024-04-26T20:27:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.204:6834/3492275136,v1:172.21.15.204:6835/3492275136] up:boot 2024-04-26T20:27:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.103:6836/1144594115,v1:172.21.15.103:6839/1144594115] up:boot 2024-04-26T20:27:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.204:6836/1832012233,v1:172.21.15.204:6838/1832012233] up:boot 2024-04-26T20:27:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.204:6837/1175426508,v1:172.21.15.204:6839/1175426508] up:boot 2024-04-26T20:27:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: mds.? [v2:172.21.15.94:6840/3605429688,v1:172.21.15.94:6841/3605429688] up:boot 2024-04-26T20:27:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: fsmap 12 up:standby 2024-04-26T20:27:42.785 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:27:42.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:27:42.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:27:42.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:27:42.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:27:42.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: from='mgr.14152 172.21.15.94:0/3992850632' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T20:27:42.786 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:42 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2018132148' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:42.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.809+0000 7f4f3ea00640 1 -- 172.21.15.94:0/1334767381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38109fc0 msgr2=0x7f4f3810a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:42.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.809+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/1334767381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38109fc0 0x7f4f3810a440 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009860 tx=0x7f4f2402a340 comp rx=0 tx=0).stop 2024-04-26T20:27:42.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.809+0000 7f4f3ea00640 1 -- 172.21.15.94:0/1334767381 shutdown_connections 2024-04-26T20:27:42.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.809+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/1334767381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f3810a980 0x7f4f381112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.809+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/1334767381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38109fc0 0x7f4f3810a440 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.809+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/1334767381 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f38108e70 0x7f4f38109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.809+0000 7f4f3ea00640 1 -- 172.21.15.94:0/1334767381 >> 172.21.15.94:0/1334767381 conn(0x7f4f38077e00 msgr2=0x7f4f3807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:42.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.809+0000 7f4f3ea00640 1 -- 172.21.15.94:0/1334767381 shutdown_connections 2024-04-26T20:27:42.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.809+0000 7f4f3ea00640 1 -- 172.21.15.94:0/1334767381 wait complete. 2024-04-26T20:27:42.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.811+0000 7f4f3ea00640 1 Processor -- start 2024-04-26T20:27:42.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.811+0000 7f4f3ea00640 1 -- start start 2024-04-26T20:27:42.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.811+0000 7f4f3ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38108e70 0x7f4f3819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:42.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.811+0000 7f4f3ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f38109fc0 0x7f4f3819f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:42.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.811+0000 7f4f3ea00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f3810a980 0x7f4f381a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:42.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.811+0000 7f4f3ea00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f4f381149a0 con 0x7f4f3810a980 2024-04-26T20:27:42.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.811+0000 7f4f3ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4f38114820 con 0x7f4f38108e70 2024-04-26T20:27:42.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.811+0000 7f4f3ea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4f38114b20 con 0x7f4f38109fc0 2024-04-26T20:27:42.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38108e70 0x7f4f3819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:42.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3cc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f38109fc0 0x7f4f3819f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:42.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38108e70 0x7f4f3819f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:43868/0 (socket says 172.21.15.94:43868) 2024-04-26T20:27:42.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3d600640 1 -- 172.21.15.94:0/3145451575 learned_addr learned my addr 172.21.15.94:0/3145451575 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:42.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3e000640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f3810a980 0x7f4f381a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:42.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3d600640 1 -- 172.21.15.94:0/3145451575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f38109fc0 msgr2=0x7f4f3819f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:42.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3d600640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f38109fc0 0x7f4f3819f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3d600640 1 -- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f3810a980 msgr2=0x7f4f381a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:42.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3d600640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f3810a980 0x7f4f381a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:42.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3d600640 1 -- 172.21.15.94:0/3145451575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f381a6c90 con 0x7f4f38108e70 2024-04-26T20:27:42.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.812+0000 7f4f3d600640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38108e70 0x7f4f3819f070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00e9c0 tx=0x7f4f2c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:42.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.813+0000 7f4f36000640 1 -- 172.21.15.94:0/3145451575 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c00cde0 con 0x7f4f38108e70 2024-04-26T20:27:42.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.813+0000 7f4f36000640 1 -- 172.21.15.94:0/3145451575 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f2c0035f0 con 0x7f4f38108e70 2024-04-26T20:27:42.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.813+0000 7f4f36000640 1 -- 172.21.15.94:0/3145451575 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c010690 con 0x7f4f38108e70 2024-04-26T20:27:42.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.813+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f381a6f80 con 0x7f4f38108e70 2024-04-26T20:27:42.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.813+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f381a73f0 con 0x7f4f38108e70 2024-04-26T20:27:42.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.814+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f38110200 con 0x7f4f38108e70 2024-04-26T20:27:42.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.816+0000 7f4f36000640 1 -- 172.21.15.94:0/3145451575 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4f2c004000 con 0x7f4f38108e70 2024-04-26T20:27:42.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.817+0000 7f4f36000640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4f200a3cc0 0x7f4f200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:42.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.818+0000 7f4f36000640 1 -- 172.21.15.94:0/3145451575 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8912+0+0 (secure 0 0 0) 0x7f4f2c0d0b00 con 0x7f4f38108e70 2024-04-26T20:27:42.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.818+0000 7f4f3cc00640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4f200a3cc0 0x7f4f200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:42.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.818+0000 7f4f3cc00640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4f200a3cc0 0x7f4f200a6180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009830 tx=0x7f4f24030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:42.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:42.822+0000 7f4f36000640 1 -- 172.21.15.94:0/3145451575 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4f2c009510 con 0x7f4f38108e70 2024-04-26T20:27:43.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.024+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4f3802dad0 con 0x7f4f38108e70 2024-04-26T20:27:43.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.593+0000 7f4f36000640 1 -- 172.21.15.94:0/3145451575 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) ==== 169+0+0 (secure 0 0 0) 0x7f4f2c09d0a0 con 0x7f4f38108e70 2024-04-26T20:27:43.595 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-04-26T20:27:43.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4f200a3cc0 msgr2=0x7f4f200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:43.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4f200a3cc0 0x7f4f200a6180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009830 tx=0x7f4f24030450 comp rx=0 tx=0).stop 2024-04-26T20:27:43.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38108e70 msgr2=0x7f4f3819f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:43.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38108e70 0x7f4f3819f070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00e9c0 tx=0x7f4f2c00ee80 comp rx=0 tx=0).stop 2024-04-26T20:27:43.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 shutdown_connections 2024-04-26T20:27:43.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f4f200a3cc0 0x7f4f200a6180 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:43.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f3810a980 0x7f4f381a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:43.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f38109fc0 0x7f4f3819f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:43.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 --2- 172.21.15.94:0/3145451575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f38108e70 0x7f4f3819f070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:43.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 >> 172.21.15.94:0/3145451575 conn(0x7f4f38077e00 msgr2=0x7f4f3810d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:43.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 shutdown_connections 2024-04-26T20:27:43.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.600+0000 7f4f3ea00640 1 -- 172.21.15.94:0/3145451575 wait complete. 2024-04-26T20:27:43.625 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-26T20:27:43.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.831+0000 7f565fc00640 1 -- 172.21.15.94:0/197090663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5658103b90 msgr2=0x7f5658104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:43.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.831+0000 7f565fc00640 1 --2- 172.21.15.94:0/197090663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5658103b90 0x7f5658104010 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5644009860 tx=0x7f564402a340 comp rx=0 tx=0).stop 2024-04-26T20:27:43.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.831+0000 7f565fc00640 1 -- 172.21.15.94:0/197090663 shutdown_connections 2024-04-26T20:27:43.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.831+0000 7f565fc00640 1 --2- 172.21.15.94:0/197090663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5658104550 0x7f565810ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:43.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.831+0000 7f565fc00640 1 --2- 172.21.15.94:0/197090663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5658103b90 0x7f5658104010 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:43.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.831+0000 7f565fc00640 1 --2- 172.21.15.94:0/197090663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56581029a0 0x7f5658102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:43.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.831+0000 7f565fc00640 1 -- 172.21.15.94:0/197090663 >> 172.21.15.94:0/197090663 conn(0x7f56580fdef0 msgr2=0x7f5658100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:43.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.831+0000 7f565fc00640 1 -- 172.21.15.94:0/197090663 shutdown_connections 2024-04-26T20:27:43.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.831+0000 7f565fc00640 1 -- 172.21.15.94:0/197090663 wait complete. 2024-04-26T20:27:43.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.832+0000 7f565fc00640 1 Processor -- start 2024-04-26T20:27:43.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.832+0000 7f565fc00640 1 -- start start 2024-04-26T20:27:43.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565fc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56581029a0 0x7f565819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565fc00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5658103b90 0x7f565819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565fc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5658104550 0x7f56581a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565fc00640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 -- 0x7f565810e4b0 con 0x7f5658103b90 2024-04-26T20:27:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565fc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f565810e330 con 0x7f56581029a0 2024-04-26T20:27:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565fc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f565810e630 con 0x7f5658104550 2024-04-26T20:27:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f5657e00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5658103b90 0x7f565819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:43.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565d200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56581029a0 0x7f565819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:43.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565d200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56581029a0 0x7f565819adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.94:43896/0 (socket says 172.21.15.94:43896) 2024-04-26T20:27:43.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565d200640 1 -- 172.21.15.94:0/728583657 learned_addr learned my addr 172.21.15.94:0/728583657 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-04-26T20:27:43.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f565dc00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5658104550 0x7f56581a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:43.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f5657e00640 1 -- 172.21.15.94:0/728583657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5658104550 msgr2=0x7f56581a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:43.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.833+0000 7f5657e00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5658104550 0x7f56581a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:43.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.834+0000 7f5657e00640 1 -- 172.21.15.94:0/728583657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56581029a0 msgr2=0x7f565819adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:43.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.834+0000 7f5657e00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56581029a0 0x7f565819adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:43.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.834+0000 7f5657e00640 1 -- 172.21.15.94:0/728583657 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56581a29d0 con 0x7f5658103b90 2024-04-26T20:27:43.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.834+0000 7f5657e00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5658103b90 0x7f565819b300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f564402a860 tx=0x7f5644032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:43.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.834+0000 7f5655600640 1 -- 172.21.15.94:0/728583657 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644037040 con 0x7f5658103b90 2024-04-26T20:27:43.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.834+0000 7f5655600640 1 -- 172.21.15.94:0/728583657 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5644030bd0 con 0x7f5658103b90 2024-04-26T20:27:43.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.834+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56581a2c60 con 0x7f5658103b90 2024-04-26T20:27:43.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.834+0000 7f5655600640 1 -- 172.21.15.94:0/728583657 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644003850 con 0x7f5658103b90 2024-04-26T20:27:43.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.834+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56581a3140 con 0x7f5658103b90 2024-04-26T20:27:43.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.835+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56581a3310 con 0x7f5658103b90 2024-04-26T20:27:43.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.837+0000 7f5655600640 1 -- 172.21.15.94:0/728583657 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5644043020 con 0x7f5658103b90 2024-04-26T20:27:43.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.839+0000 7f5655600640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f56400a3cc0 0x7f56400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:43.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.839+0000 7f5655600640 1 -- 172.21.15.94:0/728583657 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8324+0+0 (secure 0 0 0) 0x7f56440ef600 con 0x7f5658103b90 2024-04-26T20:27:43.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.839+0000 7f565d200640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f56400a3cc0 0x7f56400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:43.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.840+0000 7f565d200640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f56400a3cc0 0x7f56400a6180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f565819c290 tx=0x7f564c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:43.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:43.843+0000 7f5655600640 1 -- 172.21.15.94:0/728583657 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f56440bbed0 con 0x7f5658103b90 2024-04-26T20:27:43.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:43 smithi204 ceph-mon[32618]: pgmap v912: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-26T20:27:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:43 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/2018132148' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:27:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:43 smithi204 ceph-mon[32618]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T20:27:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:43 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/3145451575' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:43.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:43 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:43.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:43 smithi094 ceph-mon[28050]: pgmap v912: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-26T20:27:43.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:43 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/2018132148' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:27:43.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:43 smithi094 ceph-mon[28050]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T20:27:43.987 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:43 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/3145451575' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:43.988 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:43 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:44.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:43 smithi103 ceph-mon[32635]: pgmap v912: 129 pgs: 129 active+clean; 642 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-26T20:27:44.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:43 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/2018132148' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:27:44.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:43 smithi103 ceph-mon[32635]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T20:27:44.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:43 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/3145451575' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:44.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:43 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:44.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.050+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f56581a3c50 con 0x7f5658103b90 2024-04-26T20:27:44.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.600+0000 7f5655600640 1 -- 172.21.15.94:0/728583657 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v100) ==== 178+0+0 (secure 0 0 0) 0x7f5644030d80 con 0x7f5658103b90 2024-04-26T20:27:44.602 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data_ec' removed 2024-04-26T20:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.607+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f56400a3cc0 msgr2=0x7f56400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.607+0000 7f565fc00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f56400a3cc0 0x7f56400a6180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f565819c290 tx=0x7f564c009260 comp rx=0 tx=0).stop 2024-04-26T20:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.607+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5658103b90 msgr2=0x7f565819b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.607+0000 7f565fc00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5658103b90 0x7f565819b300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f564402a860 tx=0x7f5644032dc0 comp rx=0 tx=0).stop 2024-04-26T20:27:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.609+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 shutdown_connections 2024-04-26T20:27:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.609+0000 7f565fc00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:6800/2112565670,v1:172.21.15.94:6801/2112565670] conn(0x7f56400a3cc0 0x7f56400a6180 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.609+0000 7f565fc00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5658104550 0x7f56581a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.609+0000 7f565fc00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5658103b90 0x7f565819b300 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.609+0000 7f565fc00640 1 --2- 172.21.15.94:0/728583657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56581029a0 0x7f565819adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.609+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 >> 172.21.15.94:0/728583657 conn(0x7f56580fdef0 msgr2=0x7f5658109330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.609+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 shutdown_connections 2024-04-26T20:27:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-04-26T20:27:44.609+0000 7f565fc00640 1 -- 172.21.15.94:0/728583657 wait complete. 2024-04-26T20:27:44.636 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:27:44.700 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:27:44.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:27:44.764 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T20:27:44.765 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T20:27:44.823 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T20:27:44.841 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T20:27:44.858 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T20:27:44.858 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T20:27:44.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.a 2024-04-26T20:27:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:44 smithi204 ceph-mon[32618]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:27:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:44 smithi204 ceph-mon[32618]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T20:27:44.881 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:44 smithi204 ceph-mon[32618]: from='client.? 172.21.15.94:0/728583657' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:44.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:44 smithi094 ceph-mon[28050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:27:44.926 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:44 smithi094 ceph-mon[28050]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T20:27:44.927 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:44 smithi094 ceph-mon[28050]: from='client.? 172.21.15.94:0/728583657' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:45.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:44 smithi103 ceph-mon[32635]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:27:45.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:44 smithi103 ceph-mon[32635]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T20:27:45.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:44 smithi103 ceph-mon[32635]: from='client.? 172.21.15.94:0/728583657' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:27:45.237 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:44 smithi094 systemd[1]: Stopping Ceph mon.a for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:27:45.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a[28046]: 2024-04-26T20:27:45.457+0000 7f5f5f800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:27:45.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a[28046]: 2024-04-26T20:27:45.457+0000 7f5f5f800700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T20:27:45.738 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:45 smithi094 podman[82844]: 2024-04-26 20:27:45.540738624 +0000 UTC m=+0.251015945 container died 276d9903869a5947568af1c726803b658a8d96c9f563226a2a912c212472f712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T20:27:46.437 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:46 smithi094 podman[82844]: 2024-04-26 20:27:46.126462308 +0000 UTC m=+0.836739606 container cleanup 276d9903869a5947568af1c726803b658a8d96c9f563226a2a912c212472f712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-26T20:27:46.437 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:46 smithi094 bash[82844]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a 2024-04-26T20:27:46.689 INFO:journalctl@ceph.mon.a.smithi094.stdout:Apr 26 20:27:46 smithi094 podman[82858]: 2024-04-26 20:27:46.436564841 +0000 UTC m=+0.882437201 container remove 276d9903869a5947568af1c726803b658a8d96c9f563226a2a912c212472f712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T20:27:46.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T20:27:46.854 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:27:46.854 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T20:27:46.854 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-26T20:27:46.855 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.b 2024-04-26T20:27:47.154 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:46 smithi103 systemd[1]: Stopping Ceph mon.b for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:27:47.437 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-b[32631]: 2024-04-26T20:27:47.152+0000 7f384b072700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:27:47.437 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-b[32631]: 2024-04-26T20:27:47.152+0000 7f384b072700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T20:27:47.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:47 smithi103 podman[67762]: 2024-04-26 20:27:47.224431233 +0000 UTC m=+0.167296278 container died e6e36d2770df6204ac3ac1bec0532d50847634944f51f22deb8d2b20af003eb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T20:27:47.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:47 smithi103 podman[67762]: 2024-04-26 20:27:47.611895269 +0000 UTC m=+0.554760315 container cleanup e6e36d2770df6204ac3ac1bec0532d50847634944f51f22deb8d2b20af003eb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-b, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:27:47.784 INFO:journalctl@ceph.mon.b.smithi103.stdout:Apr 26 20:27:47 smithi103 bash[67762]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-b 2024-04-26T20:27:48.021 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T20:27:48.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:27:48.064 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-26T20:27:48.065 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-26T20:27:48.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.c 2024-04-26T20:27:48.381 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:48 smithi204 systemd[1]: Stopping Ceph mon.c for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:27:48.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:48 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-c[32600]: 2024-04-26T20:27:48.526+0000 7fe15c9c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:27:48.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:48 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-c[32600]: 2024-04-26T20:27:48.526+0000 7fe15c9c5700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T20:27:48.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:48 smithi204 podman[68084]: 2024-04-26 20:27:48.618699181 +0000 UTC m=+0.356583517 container died f5675512b043ca5046878e4713a6039a92ed2a21048a171219c79259db1227c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:27:49.288 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:49 smithi204 podman[68084]: 2024-04-26 20:27:49.086368535 +0000 UTC m=+0.824252892 container cleanup f5675512b043ca5046878e4713a6039a92ed2a21048a171219c79259db1227c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:27:49.288 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 26 20:27:49 smithi204 bash[68084]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mon-c 2024-04-26T20:27:49.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T20:27:49.558 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:27:49.558 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-26T20:27:49.558 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T20:27:49.559 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.x 2024-04-26T20:27:49.987 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:27:49 smithi094 systemd[1]: Stopping Ceph mgr.x for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:27:50.488 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:27:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:27:50.204+0000 7f0311a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:27:50.488 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:27:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x[28285]: 2024-04-26T20:27:50.204+0000 7f0311a00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T20:27:50.488 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:27:50 smithi094 podman[82965]: 2024-04-26 20:27:50.297441236 +0000 UTC m=+0.284216253 container died dd64914477745935a123fae9e5fc59f4a263a11de5f5042830ffd2a3b463084c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:27:50.987 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:27:50 smithi094 podman[82965]: 2024-04-26 20:27:50.823669863 +0000 UTC m=+0.810444899 container cleanup dd64914477745935a123fae9e5fc59f4a263a11de5f5042830ffd2a3b463084c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T20:27:50.988 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:27:50 smithi094 bash[82965]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x 2024-04-26T20:27:51.462 INFO:journalctl@ceph.mgr.x.smithi094.stdout:Apr 26 20:27:51 smithi094 podman[82978]: 2024-04-26 20:27:51.051618186 +0000 UTC m=+0.740761332 container remove dd64914477745935a123fae9e5fc59f4a263a11de5f5042830ffd2a3b463084c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T20:27:51.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T20:27:51.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:27:51.604 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T20:27:51.604 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T20:27:51.604 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.y 2024-04-26T20:27:51.938 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:27:51 smithi103 systemd[1]: Stopping Ceph mgr.y for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:27:52.260 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:27:51 smithi103 podman[67886]: 2024-04-26 20:27:51.937333341 +0000 UTC m=+0.140244360 container died d8d8492861488bdd01b4dd911f841ccc8ad1187e2565523789984b04919362ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:27:52.534 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:27:52 smithi103 podman[67886]: 2024-04-26 20:27:52.351035733 +0000 UTC m=+0.553946754 container cleanup d8d8492861488bdd01b4dd911f841ccc8ad1187e2565523789984b04919362ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T20:27:52.534 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Apr 26 20:27:52 smithi103 bash[67886]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-y 2024-04-26T20:27:52.693 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T20:27:52.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:27:52.727 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T20:27:52.727 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-26T20:27:52.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.z 2024-04-26T20:27:53.131 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:27:52 smithi204 systemd[1]: Stopping Ceph mgr.z for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:27:53.418 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:27:53 smithi204 podman[68205]: 2024-04-26 20:27:53.137275988 +0000 UTC m=+0.221000863 container died aa57009e2a3cd344407767d22ee94fe57d4ed722ba251ed3c03773ef77e0ebd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T20:27:53.836 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:27:53 smithi204 podman[68205]: 2024-04-26 20:27:53.49221077 +0000 UTC m=+0.575935644 container cleanup aa57009e2a3cd344407767d22ee94fe57d4ed722ba251ed3c03773ef77e0ebd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:27:53.836 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 26 20:27:53 smithi204 bash[68205]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mgr-z 2024-04-26T20:27:54.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-26T20:27:54.083 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:27:54.084 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-26T20:27:54.084 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T20:27:54.084 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.0 2024-04-26T20:27:54.407 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:27:54 smithi094 systemd[1]: Stopping Ceph osd.0 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:27:54.737 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:27:54 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0[40056]: 2024-04-26T20:27:54.642+0000 7f7636800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:27:54.737 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:27:54 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0[40056]: 2024-04-26T20:27:54.642+0000 7f7636800700 -1 osd.0 100 *** Got signal Terminated *** 2024-04-26T20:27:54.738 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:27:54 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0[40056]: 2024-04-26T20:27:54.642+0000 7f7636800700 -1 osd.0 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:28:00.237 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:27:59 smithi094 podman[83099]: 2024-04-26 20:27:59.978439988 +0000 UTC m=+5.486713353 container died 09ebc3c5b3638024cea3dc7d142be2d099707128ea4d4d71eb6e1ab8e8cca845 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T20:28:00.716 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:28:00 smithi094 podman[83099]: 2024-04-26 20:28:00.471362013 +0000 UTC m=+5.979635407 container cleanup 09ebc3c5b3638024cea3dc7d142be2d099707128ea4d4d71eb6e1ab8e8cca845 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T20:28:00.716 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:28:00 smithi094 bash[83099]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0 2024-04-26T20:28:00.987 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:28:00 smithi094 podman[83116]: 2024-04-26 20:28:00.714506559 +0000 UTC m=+0.714324398 container remove 09ebc3c5b3638024cea3dc7d142be2d099707128ea4d4d71eb6e1ab8e8cca845 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:28:01.912 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:28:01 smithi094 podman[83209]: 2024-04-26 20:28:01.433443363 +0000 UTC m=+0.036035266 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:28:01.912 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:28:01 smithi094 podman[83209]: 2024-04-26 20:28:01.565664802 +0000 UTC m=+0.168256756 container create 2e86d4021d4c47cbb4fa7f8dd5bffa1f93d17d7083340953cf4e189e17b5820d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:28:02.237 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:28:01 smithi094 podman[83209]: 2024-04-26 20:28:01.911559653 +0000 UTC m=+0.514151625 container init 2e86d4021d4c47cbb4fa7f8dd5bffa1f93d17d7083340953cf4e189e17b5820d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-26T20:28:02.237 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:28:01 smithi094 podman[83209]: 2024-04-26 20:28:01.923730554 +0000 UTC m=+0.526322453 container start 2e86d4021d4c47cbb4fa7f8dd5bffa1f93d17d7083340953cf4e189e17b5820d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:28:02.238 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:28:02 smithi094 podman[83209]: 2024-04-26 20:28:02.027478291 +0000 UTC m=+0.630070269 container attach 2e86d4021d4c47cbb4fa7f8dd5bffa1f93d17d7083340953cf4e189e17b5820d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:28:02.721 INFO:journalctl@ceph.osd.0.smithi094.stdout:Apr 26 20:28:02 smithi094 podman[83209]: 2024-04-26 20:28:02.366757383 +0000 UTC m=+0.969349334 container died 2e86d4021d4c47cbb4fa7f8dd5bffa1f93d17d7083340953cf4e189e17b5820d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-26T20:28:03.186 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T20:28:03.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:28:03.303 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T20:28:03.303 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T20:28:03.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.1 2024-04-26T20:28:03.606 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:03 smithi103 systemd[1]: Stopping Ceph osd.1 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:28:04.034 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:03 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1[37217]: 2024-04-26T20:28:03.605+0000 7fd9d6746700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:28:04.034 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:03 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1[37217]: 2024-04-26T20:28:03.605+0000 7fd9d6746700 -1 osd.1 100 *** Got signal Terminated *** 2024-04-26T20:28:04.035 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:03 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1[37217]: 2024-04-26T20:28:03.605+0000 7fd9d6746700 -1 osd.1 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:28:09.466 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:09 smithi103 podman[68027]: 2024-04-26 20:28:09.146517442 +0000 UTC m=+5.640842580 container died 664e5e84dd1348d6f5549bc24fc1e188fa33e56f57359e33d076790ccfdf9600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T20:28:09.784 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:09 smithi103 podman[68027]: 2024-04-26 20:28:09.564456981 +0000 UTC m=+6.058782119 container cleanup 664e5e84dd1348d6f5549bc24fc1e188fa33e56f57359e33d076790ccfdf9600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:28:09.784 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:09 smithi103 bash[68027]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1 2024-04-26T20:28:10.241 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:09 smithi103 podman[68044]: 2024-04-26 20:28:09.807560755 +0000 UTC m=+0.654563781 container remove 664e5e84dd1348d6f5549bc24fc1e188fa33e56f57359e33d076790ccfdf9600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T20:28:10.534 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:10 smithi103 podman[68135]: 2024-04-26 20:28:10.143909967 +0000 UTC m=+0.018903108 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:28:10.534 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:10 smithi103 podman[68135]: 2024-04-26 20:28:10.250326494 +0000 UTC m=+0.125319614 container create e17db38c977823b0bdbb69613ed34d2326be5463290e97fef876362e2f18a3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T20:28:10.534 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:10 smithi103 podman[68135]: 2024-04-26 20:28:10.477792137 +0000 UTC m=+0.352785256 container init e17db38c977823b0bdbb69613ed34d2326be5463290e97fef876362e2f18a3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-04-26T20:28:10.535 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:10 smithi103 podman[68135]: 2024-04-26 20:28:10.480924205 +0000 UTC m=+0.355917324 container start e17db38c977823b0bdbb69613ed34d2326be5463290e97fef876362e2f18a3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T20:28:11.034 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:10 smithi103 podman[68135]: 2024-04-26 20:28:10.578114888 +0000 UTC m=+0.453108016 container attach e17db38c977823b0bdbb69613ed34d2326be5463290e97fef876362e2f18a3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-04-26T20:28:11.034 INFO:journalctl@ceph.osd.1.smithi103.stdout:Apr 26 20:28:10 smithi103 podman[68135]: 2024-04-26 20:28:10.784133896 +0000 UTC m=+0.659127019 container died e17db38c977823b0bdbb69613ed34d2326be5463290e97fef876362e2f18a3cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:28:11.434 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T20:28:11.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:28:11.475 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T20:28:11.475 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T20:28:11.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.2 2024-04-26T20:28:11.831 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:11 smithi204 systemd[1]: Stopping Ceph osd.2 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:28:12.131 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:11 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2[37758]: 2024-04-26T20:28:11.830+0000 7f9fcb789700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:28:12.132 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:11 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2[37758]: 2024-04-26T20:28:11.830+0000 7f9fcb789700 -1 osd.2 100 *** Got signal Terminated *** 2024-04-26T20:28:12.132 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:11 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2[37758]: 2024-04-26T20:28:11.830+0000 7f9fcb789700 -1 osd.2 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:28:17.526 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:17 smithi204 podman[68346]: 2024-04-26 20:28:17.232721455 +0000 UTC m=+5.566994957 container died dbf968be4d73f839f1c02fbd472ee3f34d5f313f2e9e43dddc7f81cc62bc1d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:28:17.881 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:17 smithi204 podman[68346]: 2024-04-26 20:28:17.625209514 +0000 UTC m=+5.959483004 container cleanup dbf968be4d73f839f1c02fbd472ee3f34d5f313f2e9e43dddc7f81cc62bc1d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:28:17.882 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:17 smithi204 bash[68346]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2 2024-04-26T20:28:18.269 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:17 smithi204 podman[68363]: 2024-04-26 20:28:17.959936698 +0000 UTC m=+0.721651230 container remove dbf968be4d73f839f1c02fbd472ee3f34d5f313f2e9e43dddc7f81cc62bc1d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True) 2024-04-26T20:28:18.547 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:18 smithi204 podman[68460]: 2024-04-26 20:28:18.269229718 +0000 UTC m=+0.093067509 container create ab2eb0cbcda2f45b1f2269288d21cce5d849320189bb8d6a3d7ed0258d3b0e22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True) 2024-04-26T20:28:18.548 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:18 smithi204 podman[68460]: 2024-04-26 20:28:18.194000685 +0000 UTC m=+0.017838504 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:28:18.548 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:18 smithi204 podman[68460]: 2024-04-26 20:28:18.547494299 +0000 UTC m=+0.371332091 container init ab2eb0cbcda2f45b1f2269288d21cce5d849320189bb8d6a3d7ed0258d3b0e22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-26T20:28:18.850 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:18 smithi204 podman[68460]: 2024-04-26 20:28:18.550899351 +0000 UTC m=+0.374737152 container start ab2eb0cbcda2f45b1f2269288d21cce5d849320189bb8d6a3d7ed0258d3b0e22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:28:18.851 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:18 smithi204 podman[68460]: 2024-04-26 20:28:18.655638548 +0000 UTC m=+0.479476347 container attach ab2eb0cbcda2f45b1f2269288d21cce5d849320189bb8d6a3d7ed0258d3b0e22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-04-26T20:28:19.131 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 26 20:28:18 smithi204 podman[68460]: 2024-04-26 20:28:18.850802933 +0000 UTC m=+0.674640735 container died ab2eb0cbcda2f45b1f2269288d21cce5d849320189bb8d6a3d7ed0258d3b0e22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T20:28:19.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T20:28:19.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:28:19.480 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T20:28:19.481 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T20:28:19.481 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.3 2024-04-26T20:28:19.988 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:19 smithi094 systemd[1]: Stopping Ceph osd.3 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:28:20.488 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:20 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3[46558]: 2024-04-26T20:28:20.041+0000 7fc4c1200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:28:20.488 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:20 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3[46558]: 2024-04-26T20:28:20.041+0000 7fc4c1200700 -1 osd.3 100 *** Got signal Terminated *** 2024-04-26T20:28:20.488 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:20 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3[46558]: 2024-04-26T20:28:20.041+0000 7fc4c1200700 -1 osd.3 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:28:21.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:20 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:20.765+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:22.117 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:21 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:21.809+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:23.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:22 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:22.830+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:23.487 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:23 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3[46558]: 2024-04-26T20:28:23.095+0000 7fc4bca00700 -1 osd.3 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.861787+0000 front 2024-04-26T20:27:57.861554+0000 (oldest deadline 2024-04-26T20:28:22.560738+0000) 2024-04-26T20:28:23.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:23 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:23.386+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:23.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:23 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:23.558+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:24.237 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:24 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3[46558]: 2024-04-26T20:28:24.129+0000 7fc4bca00700 -1 osd.3 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.861787+0000 front 2024-04-26T20:27:57.861554+0000 (oldest deadline 2024-04-26T20:28:22.560738+0000) 2024-04-26T20:28:24.237 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:24 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:24.025+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:24.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:23 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:23.810+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:24.881 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:24.419+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:24.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:24.607+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:25.237 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:25 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:25.002+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:25.243 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:24 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:24.788+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:25.852 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:25 smithi094 podman[83313]: 2024-04-26 20:28:25.498383235 +0000 UTC m=+5.604256873 container died 3692f4ec0463d0140d9c6276f70398cca24a9618c80e7eab819131760b27400a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:28:25.881 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:25 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:25.401+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:25.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:25 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:25.562+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:26.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:25 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:25.745+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:26.204 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:25 smithi094 podman[83313]: 2024-04-26 20:28:25.977333788 +0000 UTC m=+6.083207443 container cleanup 3692f4ec0463d0140d9c6276f70398cca24a9618c80e7eab819131760b27400a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:28:26.205 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:25 smithi094 bash[83313]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3 2024-04-26T20:28:26.205 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:26 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:26.039+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:26.487 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:26 smithi094 podman[83344]: 2024-04-26 20:28:26.20357459 +0000 UTC m=+0.686460933 container remove 3692f4ec0463d0140d9c6276f70398cca24a9618c80e7eab819131760b27400a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T20:28:26.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:26 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:26.583+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:26.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:26 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:26.404+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:27.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:26 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:26.777+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:27.261 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:27 smithi094 podman[83433]: 2024-04-26 20:28:26.914066141 +0000 UTC m=+0.037987446 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:28:27.261 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:27 smithi094 podman[83433]: 2024-04-26 20:28:27.071561978 +0000 UTC m=+0.195483306 container create ae7c6f8df62f7143e03205ea16d9bc2848d26947040f124ef672822a9361ab8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-26T20:28:27.262 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:27 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:27.016+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:27.631 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:27.371+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:27.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:27.604+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:27.737 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:27 smithi094 podman[83433]: 2024-04-26 20:28:27.377851437 +0000 UTC m=+0.501772795 container init ae7c6f8df62f7143e03205ea16d9bc2848d26947040f124ef672822a9361ab8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:28:27.737 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:27 smithi094 podman[83433]: 2024-04-26 20:28:27.390115831 +0000 UTC m=+0.514037196 container start ae7c6f8df62f7143e03205ea16d9bc2848d26947040f124ef672822a9361ab8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T20:28:27.737 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:27 smithi094 podman[83433]: 2024-04-26 20:28:27.49143457 +0000 UTC m=+0.615355878 container attach ae7c6f8df62f7143e03205ea16d9bc2848d26947040f124ef672822a9361ab8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:28:28.225 INFO:journalctl@ceph.osd.3.smithi094.stdout:Apr 26 20:28:27 smithi094 podman[83433]: 2024-04-26 20:28:27.841649779 +0000 UTC m=+0.965571124 container died ae7c6f8df62f7143e03205ea16d9bc2848d26947040f124ef672822a9361ab8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T20:28:28.226 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:28 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:28.058+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:28.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:27 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:27.825+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:28.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T20:28:28.652 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:28 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:28.400+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:28.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:28:28.726 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T20:28:28.726 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T20:28:28.726 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.4 2024-04-26T20:28:29.026 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:28 smithi103 systemd[1]: Stopping Ceph osd.4 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:28:29.027 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:28 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:28.787+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:29.131 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:28 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:28.651+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:29.284 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:29 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4[43736]: 2024-04-26T20:28:29.026+0000 7f75da34c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:28:29.284 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:29 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4[43736]: 2024-04-26T20:28:29.026+0000 7f75da34c700 -1 osd.4 100 *** Got signal Terminated *** 2024-04-26T20:28:29.284 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:29 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4[43736]: 2024-04-26T20:28:29.026+0000 7f75da34c700 -1 osd.4 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:28:29.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:29 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:29.014+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:29.628 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:29.365+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:29.629 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:29.365+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:29.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:29.627+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:30.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:29 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:29.771+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:30.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:29 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:29.771+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:30.381 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:30.356+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:30.382 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:30.356+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:30.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:29 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:29.987+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:30.784 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:30 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:30.456+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:30.784 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:30 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4[43736]: 2024-04-26T20:28:30.472+0000 7f75d6940700 -1 osd.4 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836584+0000 front 2024-04-26T20:28:06.836608+0000 (oldest deadline 2024-04-26T20:28:29.736759+0000) 2024-04-26T20:28:30.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:30.595+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:30.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:30.602+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:30.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:30.602+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:31.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:30 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:30.793+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:31.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:30 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:30.793+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:31.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:30 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:30.996+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:31.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:30 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:30.996+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:31.559 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:31.372+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:31.560 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:31.372+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:31.783 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:31 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:31.498+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:31.783 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:31 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4[43736]: 2024-04-26T20:28:31.492+0000 7f75d6940700 -1 osd.4 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836584+0000 front 2024-04-26T20:28:06.836608+0000 (oldest deadline 2024-04-26T20:28:29.736759+0000) 2024-04-26T20:28:31.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:31.558+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:31.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:31.602+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:31.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:31.602+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:32.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:31 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:31.782+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:32.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:31 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:31.782+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:32.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:32 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:32.001+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:32.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:32 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:32.001+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:32.631 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:32 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:32.581+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:32.632 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:32 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:32.346+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:32.632 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:32 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:32.346+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:32.784 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:32 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4[43736]: 2024-04-26T20:28:32.515+0000 7f75d6940700 -1 osd.4 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836584+0000 front 2024-04-26T20:28:06.836608+0000 (oldest deadline 2024-04-26T20:28:29.736759+0000) 2024-04-26T20:28:32.784 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:32 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:32.535+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:32.785 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:32 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:32.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:32.785 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:32 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:32.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:33.131 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:32 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:32.649+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:33.131 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:32 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:32.649+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:33.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:33 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:33.025+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:33.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:33 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:33.025+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:33.632 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:33.562+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:33.632 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:33.369+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:33.632 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:33.369+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:34.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:33 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:33.574+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:34.034 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:33 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4[43736]: 2024-04-26T20:28:33.560+0000 7f75d6940700 -1 osd.4 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836584+0000 front 2024-04-26T20:28:06.836608+0000 (oldest deadline 2024-04-26T20:28:29.736759+0000) 2024-04-26T20:28:34.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:33 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:33.771+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:34.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:33 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:33.771+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:34.131 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:33.631+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:34.131 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:33 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:33.631+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:34.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:34 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:34.067+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:34.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:34 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:34.067+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:34.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:34 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:34.279+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:34.659 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:34 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:34.565+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:34.659 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:34 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:34.394+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:34.659 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:34 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:34.394+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:34.852 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:34 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:34.558+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:34.853 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:34 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:34.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:34.853 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:34 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:34.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:34.853 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:34 smithi103 podman[68265]: 2024-04-26 20:28:34.565285496 +0000 UTC m=+5.643995045 container died 06578b1ee8247f14ff4899d5bd36e66b16da52b4a857158574b4689167798286 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T20:28:35.131 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:34 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:34.658+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:35.131 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:34 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:34.658+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:35.203 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:34 smithi103 podman[68265]: 2024-04-26 20:28:34.9601237 +0000 UTC m=+6.038833235 container cleanup 06578b1ee8247f14ff4899d5bd36e66b16da52b4a857158574b4689167798286 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T20:28:35.203 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:34 smithi103 bash[68265]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4 2024-04-26T20:28:35.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:35 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:35.239+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:35.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:35 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:35.032+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:35.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:35 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:35.032+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:35.531 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:35 smithi103 podman[68285]: 2024-04-26 20:28:35.203211346 +0000 UTC m=+0.634120258 container remove 06578b1ee8247f14ff4899d5bd36e66b16da52b4a857158574b4689167798286 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:28:35.781 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:35 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:35.570+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:35.782 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:35 smithi103 podman[68376]: 2024-04-26 20:28:35.434808241 +0000 UTC m=+0.018184456 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:28:35.782 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:35 smithi103 podman[68376]: 2024-04-26 20:28:35.537552477 +0000 UTC m=+0.120928723 container create 69b109c8d576a9d5b4d35049af833655ca0bebccba9e02af8754a8481a00a37c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:28:35.782 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:35 smithi103 podman[68376]: 2024-04-26 20:28:35.781786393 +0000 UTC m=+0.365162596 container init 69b109c8d576a9d5b4d35049af833655ca0bebccba9e02af8754a8481a00a37c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T20:28:35.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:35 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:35.569+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:35.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:35 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:35.612+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:35.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:35 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:35.612+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:35.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:35 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:35.394+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:35.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:35 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:35.394+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:36.034 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:35 smithi103 podman[68376]: 2024-04-26 20:28:35.785070267 +0000 UTC m=+0.368446479 container start 69b109c8d576a9d5b4d35049af833655ca0bebccba9e02af8754a8481a00a37c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-04-26T20:28:36.035 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:35 smithi103 podman[68376]: 2024-04-26 20:28:35.873379685 +0000 UTC m=+0.456755905 container attach 69b109c8d576a9d5b4d35049af833655ca0bebccba9e02af8754a8481a00a37c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True) 2024-04-26T20:28:36.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:35 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:35.787+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:36.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:35 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:35.787+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:36.452 INFO:journalctl@ceph.osd.4.smithi103.stdout:Apr 26 20:28:36 smithi103 podman[68376]: 2024-04-26 20:28:36.089384387 +0000 UTC m=+0.672760598 container died 69b109c8d576a9d5b4d35049af833655ca0bebccba9e02af8754a8481a00a37c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:28:36.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:36 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:36.224+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:36.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:36 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:36.049+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:36.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:36 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:36.049+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:36.656 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:36.542+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:36.657 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:36.396+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:36.657 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:36.396+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:36.760 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:36 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:36.524+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:36.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T20:28:36.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:28:36.820 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T20:28:36.821 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T20:28:36.821 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.5 2024-04-26T20:28:37.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:36 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:36.821+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:37.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:36 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:36.821+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:37.132 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:36 smithi204 systemd[1]: Stopping Ceph osd.5 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:28:37.132 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:36.655+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:37.132 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:36 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:36.655+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:37.283 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:37 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:37.254+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:37.283 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:37 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:37.020+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:37.284 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:37 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:37.020+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:37.445 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:37.146+0000 7f85759c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:28:37.445 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:37.146+0000 7f85759c1700 -1 osd.5 100 *** Got signal Terminated *** 2024-04-26T20:28:37.445 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:37.146+0000 7f85759c1700 -1 osd.5 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:28:37.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:37.512+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:37.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:37.665+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:37.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:37.665+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:37.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:37.444+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:37.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:37 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:37.444+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:38.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:37 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:37.553+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:38.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:37 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:37.801+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:38.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:37 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:37.801+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:38.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:38 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:38.258+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:38.489 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:38 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:38.030+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:38.489 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:38 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:38.030+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:38.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:38 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:38.485+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:38.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:38 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:38.418+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:38.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:38 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:38.418+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:38.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:38 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:38.640+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:38.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:38 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:38.640+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:38.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:38 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:38.640+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:39.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:38.564+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:39.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:38.805+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:39.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:38.805+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:39.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:38 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:38.805+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:39.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:39 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:39.221+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:39.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:39 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:39.221+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:39.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:39 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:39.025+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:39.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:39 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:39.025+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:39.847 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:39.554+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:39.847 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:39.554+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:39.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:39 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:39.485+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:39.881 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:39 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:39.414+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:39.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:39 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:39.414+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:39.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:39 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:39.612+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:39.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:39 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:39.612+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:39.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:39 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:39.612+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:40.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:39.846+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:40.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:39.846+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:40.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:39 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:39.846+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:40.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:40.221+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:40.488 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:40.221+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:40.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:40.015+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:40.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:40.015+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:40.631 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:40.366+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:40.632 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:40.366+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:40.632 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:40.450+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:40.632 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:40.450+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:41.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:40.590+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:41.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:40.590+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:41.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:40.822+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:41.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:40.822+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:41.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:40 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:40.822+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:41.131 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:40.646+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:41.132 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:40.646+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:41.132 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:40 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:40.646+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:41.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:41 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:41.222+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:41.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:41 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:41.222+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:41.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:40.995+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:41.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:40.995+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:41.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:40 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:40.995+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:41.606 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:41 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:41.367+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:57.747640+0000 front 2024-04-26T20:27:57.747458+0000 (oldest deadline 2024-04-26T20:28:22.447361+0000) 2024-04-26T20:28:41.606 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:41 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5[44296]: 2024-04-26T20:28:41.367+0000 7f8571fb5700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:08.248922+0000 front 2024-04-26T20:28:08.249014+0000 (oldest deadline 2024-04-26T20:28:29.349064+0000) 2024-04-26T20:28:41.607 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:41 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:41.468+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:41.607 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:41 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:41.468+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:41.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:41 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:41.605+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:41.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:41 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:41.605+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:41.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:41 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:41.605+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:42.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:41 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:41.629+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:42.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:41 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:41.629+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:42.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:41 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:41.795+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:42.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:41 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:41.795+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:42.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:41 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:41.795+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:42.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:42 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:42.220+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:42.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:42 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:42.220+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:42.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:42 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:42.035+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:42.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:42 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:42.035+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:42.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:42 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:42.035+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:42.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:42 smithi204 podman[68582]: 2024-04-26 20:28:42.60518825 +0000 UTC m=+5.585648267 container died 6e8b5837f1056f8d2318d0c57d1517bba045bf857f8bc5108b50e1949ce77d38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:28:42.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:42 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:42.499+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:42.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:42 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:42.500+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:42.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:42 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:42.643+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:42.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:42 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:42.643+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:42.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:42 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:42.643+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:43.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:42 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:42.679+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:43.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:42 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:42.679+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:43.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:42 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:42.829+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:43.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:42 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:42.829+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:43.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:42 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:42.829+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:43.183 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:42 smithi204 podman[68582]: 2024-04-26 20:28:42.989607375 +0000 UTC m=+5.970067436 container cleanup 6e8b5837f1056f8d2318d0c57d1517bba045bf857f8bc5108b50e1949ce77d38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:28:43.183 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:42 smithi204 bash[68582]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5 2024-04-26T20:28:43.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:43 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:43.183+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:43.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:43 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:43.183+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:43.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:43 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:43.037+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:43.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:43 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:43.037+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:43.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:43 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:43.037+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:43.538 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:43 smithi204 podman[68600]: 2024-04-26 20:28:43.182895821 +0000 UTC m=+0.572336457 container remove 6e8b5837f1056f8d2318d0c57d1517bba045bf857f8bc5108b50e1949ce77d38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T20:28:43.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:43 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:43.537+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:43.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:43 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:43.537+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:43.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:43 smithi204 podman[68691]: 2024-04-26 20:28:43.511602714 +0000 UTC m=+0.018148634 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:28:43.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:43 smithi204 podman[68691]: 2024-04-26 20:28:43.626072844 +0000 UTC m=+0.132618819 container create d25a6d244ced2f896cec6a6654fbe8c91ad106414348bee1ca5af55dcfb02fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T20:28:43.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:43 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:43.664+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:43.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:43 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:43.664+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:43.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:43 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:43.664+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:44.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:43 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:43.691+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:44.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:43 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:43.691+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:44.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:43 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:43.780+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:44.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:43 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:43.780+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:44.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:43 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:43.780+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:44.207 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:43 smithi204 podman[68691]: 2024-04-26 20:28:43.906915042 +0000 UTC m=+0.413460951 container init d25a6d244ced2f896cec6a6654fbe8c91ad106414348bee1ca5af55dcfb02fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T20:28:44.207 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:43 smithi204 podman[68691]: 2024-04-26 20:28:43.910148134 +0000 UTC m=+0.416694044 container start d25a6d244ced2f896cec6a6654fbe8c91ad106414348bee1ca5af55dcfb02fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T20:28:44.207 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:44 smithi204 podman[68691]: 2024-04-26 20:28:44.00404736 +0000 UTC m=+0.510593305 container attach d25a6d244ced2f896cec6a6654fbe8c91ad106414348bee1ca5af55dcfb02fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-26T20:28:44.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:44.168+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:44.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:44.168+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:44.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:44.079+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:44.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:44.079+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:44.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:44.079+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:44.504 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 26 20:28:44 smithi204 podman[68691]: 2024-04-26 20:28:44.207228299 +0000 UTC m=+0.713774219 container died d25a6d244ced2f896cec6a6654fbe8c91ad106414348bee1ca5af55dcfb02fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0) 2024-04-26T20:28:44.775 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:44 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:44.504+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:44.775 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:44 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:44.504+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:44.775 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:44 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:44.680+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:44.775 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:44 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:44.680+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:44.776 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:44 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:44.680+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:44.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T20:28:44.837 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:28:44.837 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T20:28:44.837 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T20:28:44.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.6 2024-04-26T20:28:45.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:44 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:44.713+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:45.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:44 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:44.713+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:45.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:44 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:44.812+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:45.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:44 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:44.812+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:45.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:44 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:44.812+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:45.196 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:44 smithi094 systemd[1]: Stopping Ceph osd.6 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:28:45.196 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:45.113+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:45.196 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:45.113+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:45.197 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:45.113+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:45.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:45.194+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:45.488 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:45.194+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:45.488 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:45.411+0000 7f6333a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:28:45.488 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:45.411+0000 7f6333a00700 -1 osd.6 100 *** Got signal Terminated *** 2024-04-26T20:28:45.488 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:45 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:45.411+0000 7f6333a00700 -1 osd.6 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:28:45.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:45 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:45.544+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:45.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:45 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:45.544+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:45.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:45 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:45.662+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:45.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:45 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:45.662+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:45.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:45 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:45.662+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:46.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:45 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:45.698+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:46.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:45 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:45.698+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:46.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:45 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:45.845+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:46.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:45 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:45.845+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:46.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:45 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:45.845+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:46.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:46 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:46.201+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:46.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:46 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:46.201+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:46.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:46 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:46.201+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:19.744109+0000 front 2024-04-26T20:28:19.744212+0000 (oldest deadline 2024-04-26T20:28:45.644526+0000) 2024-04-26T20:28:46.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:46 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:46.071+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:46.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:46 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:46.071+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:46.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:46 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:46.071+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:46.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:46 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:46.071+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:46.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:46 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:46.495+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:46.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:46 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:46.495+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:46.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:46 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:46.699+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:46.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:46 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:46.699+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:46.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:46 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:46.699+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:47.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:46 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:46.689+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:47.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:46 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:46.689+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:47.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:46 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:46.827+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:47.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:46 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:46.827+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:47.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:46 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:46.827+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:47.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:46 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:46.827+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:47.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:47 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:47.165+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:47.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:47 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:47.165+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:47.488 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:47 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:47.165+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:19.744109+0000 front 2024-04-26T20:28:19.744212+0000 (oldest deadline 2024-04-26T20:28:45.644526+0000) 2024-04-26T20:28:47.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:47 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:47.111+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:47.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:47 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:47.111+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:47.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:47 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:47.111+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:47.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:47 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:47.111+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:47.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:47 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:47.542+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:47.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:47 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:47.542+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:47.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:47 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:47.651+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:47.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:47 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:47.651+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:47.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:47 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:47.651+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:48.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:47.657+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:48.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:47.657+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:48.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:47.657+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:48.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:47.803+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:48.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:47.803+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:48.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:47.803+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:48.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:47 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:47.803+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:48.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:48 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:48.203+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:48.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:48 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:48.203+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:48.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:48 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:48.203+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:19.744109+0000 front 2024-04-26T20:28:19.744212+0000 (oldest deadline 2024-04-26T20:28:45.644526+0000) 2024-04-26T20:28:48.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:48 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:48.087+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:48.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:48 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:48.087+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:48.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:48 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:48.087+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:48.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:48 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:48.087+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:48.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:48 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:48.587+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:48.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:48 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:48.587+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:48.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:48 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:48.587+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:48.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:48 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:48.631+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:48.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:48 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:48.631+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:48.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:48 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:48.631+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:49.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:48 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:48.682+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:49.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:48 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:48.682+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:49.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:48 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:48.682+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:49.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:48 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:48.769+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:49.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:48 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:48.769+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:49.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:48 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:48.769+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:49.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:48 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:48.769+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:49.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:49 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:49.241+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:49.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:49 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:49.241+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:49.488 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:49 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:49.241+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:19.744109+0000 front 2024-04-26T20:28:19.744212+0000 (oldest deadline 2024-04-26T20:28:45.644526+0000) 2024-04-26T20:28:49.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:49 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:49.108+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:49.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:49 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:49.108+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:49.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:49 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:49.108+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:49.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:49 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:49.108+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:49.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:49 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:49.601+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:49.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:49 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:49.601+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:49.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:49 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:49.601+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:49.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:49 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:49.589+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:49.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:49 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:49.589+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:49.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:49 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:49.589+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:50.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:49 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:49.680+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:50.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:49 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:49.680+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:50.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:49 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:49.680+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:50.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:49 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:49.813+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:50.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:49 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:49.813+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:50.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:49 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:49.813+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:50.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:49 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:49.813+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:50.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:50.240+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.440498+0000 front 2024-04-26T20:28:07.440657+0000 (oldest deadline 2024-04-26T20:28:33.340428+0000) 2024-04-26T20:28:50.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:50.240+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.341938+0000 front 2024-04-26T20:28:13.341528+0000 (oldest deadline 2024-04-26T20:28:38.641441+0000) 2024-04-26T20:28:50.488 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6[53234]: 2024-04-26T20:28:50.240+0000 7f632a200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:19.744109+0000 front 2024-04-26T20:28:19.744212+0000 (oldest deadline 2024-04-26T20:28:45.644526+0000) 2024-04-26T20:28:50.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:50.158+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:50.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:50.158+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:50.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:50.158+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:50.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:50 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:50.158+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:50.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:50 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:50.587+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:50.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:50 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:50.587+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:50.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:50 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:50.587+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:50.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:50 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:50.597+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:50.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:50 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:50.597+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:50.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:50 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:50.597+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:51.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:50 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:50.842+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:51.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:50 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:50.842+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:51.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:50 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:50.842+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:51.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:50 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:50.842+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:51.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:50 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:50.725+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:51.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:50 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:50.725+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:51.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:50 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:50.725+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:51.128 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:50 smithi094 podman[83538]: 2024-04-26 20:28:50.844255663 +0000 UTC m=+5.565417572 container died e3e4ffa0eb63c9d1308c5dc5a8bfea34ae323395e950de8f4bdbf82145f58822 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:28:51.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:51 smithi094 podman[83538]: 2024-04-26 20:28:51.256005587 +0000 UTC m=+5.977167520 container cleanup e3e4ffa0eb63c9d1308c5dc5a8bfea34ae323395e950de8f4bdbf82145f58822 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T20:28:51.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:51 smithi094 bash[83538]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6 2024-04-26T20:28:51.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:51 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:51.181+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:51.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:51 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:51.181+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:51.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:51 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:51.181+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:51.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:51 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:51.181+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:51.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:51 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:51.553+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:51.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:51 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:51.553+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:51.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:51 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:51.553+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:51.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:51 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:51.597+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:51.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:51 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:51.597+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:51.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:51 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:51.597+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:51.987 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:51 smithi094 podman[83558]: 2024-04-26 20:28:51.582673972 +0000 UTC m=+0.727416831 container remove e3e4ffa0eb63c9d1308c5dc5a8bfea34ae323395e950de8f4bdbf82145f58822 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:28:52.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:51 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:51.713+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:52.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:51 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:51.713+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:52.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:51 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:51.713+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:52.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:51 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:51.846+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:52.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:51 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:51.846+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:52.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:51 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:51.846+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:52.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:51 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:51.846+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:52.475 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:52 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:52.175+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:52.476 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:52 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:52.175+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:52.476 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:52 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:52.175+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:52.476 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:52 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:52.175+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:52.737 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:52 smithi094 podman[83646]: 2024-04-26 20:28:52.384736141 +0000 UTC m=+0.044088174 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:28:52.737 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:52 smithi094 podman[83646]: 2024-04-26 20:28:52.475386351 +0000 UTC m=+0.134738441 container create f38688da4b0d70b0df3d0256b56d74d9d2d84b55fe91b05f85675caa679dc175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T20:28:52.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:52 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:52.552+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:52.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:52 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:52.552+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:52.883 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:52 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:52.552+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:52.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:52 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:52.589+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:52.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:52 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:52.589+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:52.884 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:52 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:52.589+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:53.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:52 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:52.763+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:53.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:52 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:52.763+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:53.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:52 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:52.763+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:53.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:52 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:52.850+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:53.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:52 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:52.850+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:53.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:52 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:52.850+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:53.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:52 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:52.850+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:53.213 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:52 smithi094 podman[83646]: 2024-04-26 20:28:52.77107412 +0000 UTC m=+0.430426234 container init f38688da4b0d70b0df3d0256b56d74d9d2d84b55fe91b05f85675caa679dc175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T20:28:53.213 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:52 smithi094 podman[83646]: 2024-04-26 20:28:52.783893946 +0000 UTC m=+0.443245977 container start f38688da4b0d70b0df3d0256b56d74d9d2d84b55fe91b05f85675caa679dc175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T20:28:53.213 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:52 smithi094 podman[83646]: 2024-04-26 20:28:52.870060384 +0000 UTC m=+0.529412451 container attach f38688da4b0d70b0df3d0256b56d74d9d2d84b55fe91b05f85675caa679dc175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T20:28:53.487 INFO:journalctl@ceph.osd.6.smithi094.stdout:Apr 26 20:28:53 smithi094 podman[83646]: 2024-04-26 20:28:53.238623417 +0000 UTC m=+0.897975454 container died f38688da4b0d70b0df3d0256b56d74d9d2d84b55fe91b05f85675caa679dc175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T20:28:53.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:53 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:53.211+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:53.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:53 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:53.211+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:53.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:53 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:53.211+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:53.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:53 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:53.211+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:53.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:53 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:53.590+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:53.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:53 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:53.590+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:53.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:53 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:53.590+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:53.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:53 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:53.546+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:53.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:53 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:53.546+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:53.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:53 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:53.546+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:53.952 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T20:28:54.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:53 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:53.749+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:54.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:53 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:53.749+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:54.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:53 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:53.749+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:54.036 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:53 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:53.837+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:54.036 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:53 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:53.837+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:54.036 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:53 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:53.837+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:54.036 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:53 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:53.837+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:54.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:28:54.046 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T20:28:54.046 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T20:28:54.046 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.7 2024-04-26T20:28:54.380 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:54 smithi103 systemd[1]: Stopping Ceph osd.7 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:28:54.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:54 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:54.212+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:54.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:54 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:54.212+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:54.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:54 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:54.212+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:54.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:54 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:54.212+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:54.762 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:54.378+0000 7f1b30179700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:28:54.762 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:54.378+0000 7f1b30179700 -1 osd.7 100 *** Got signal Terminated *** 2024-04-26T20:28:54.762 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:54.378+0000 7f1b30179700 -1 osd.7 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:28:54.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:54 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:54.595+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:54.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:54 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:54.595+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:54.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:54 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:54.595+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:54.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:54 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:54.523+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:54.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:54 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:54.523+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:54.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:54 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:54.523+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:55.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:54.759+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:55.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:54.759+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:55.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:54.759+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:55.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:54.791+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:55.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:54.791+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:55.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:54.791+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:55.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:54 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:54.791+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:55.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:55 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:55.234+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:55.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:55 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:55.234+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:55.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:55 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:55.234+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:55.488 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:55 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:55.234+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:55.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:55 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:55.578+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:55.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:55 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:55.578+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:55.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:55 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:55.578+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:55.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:55 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:55.515+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:55.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:55 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:55.515+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:55.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:55 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:55.515+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:55.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:55 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:55.515+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:28:56.285 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:55 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:55.794+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:56.285 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:55 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:55.794+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:56.285 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:55 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:55.794+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:56.285 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:55 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:55.794+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.859245+0000 front 2024-04-26T20:28:32.859347+0000 (oldest deadline 2024-04-26T20:28:55.159399+0000) 2024-04-26T20:28:56.286 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:55 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:55.787+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:56.286 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:55 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:55.787+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:56.286 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:55 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:55.787+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:56.286 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:55 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:55.787+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:56.286 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:55 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:55.787+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:28:56.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:56 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:56.272+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:56.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:56 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:56.272+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:56.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:56 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:56.272+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:56.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:56 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:56.272+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:56.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:56 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:56.603+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:56.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:56 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:56.603+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:56.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:56 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:56.603+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:56.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:56 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:56.603+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:28:56.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:56 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:56.564+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:56.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:56 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:56.564+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:56.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:56 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:56.564+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:56.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:56 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:56.564+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:28:57.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:56.802+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:57.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:56.802+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:57.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:56.802+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:57.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:56.802+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.859245+0000 front 2024-04-26T20:28:32.859347+0000 (oldest deadline 2024-04-26T20:28:55.159399+0000) 2024-04-26T20:28:57.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:56.758+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:57.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:56.758+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:57.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:56.758+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:57.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:56.758+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:57.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:56.758+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:28:57.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:57 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:57.322+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:57.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:57 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:57.322+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:57.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:57 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:57.322+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:57.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:57 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:57.322+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:57.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:57 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:57.650+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:57.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:57 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:57.650+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:57.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:57 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:57.650+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:57.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:57 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:57.650+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:28:57.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:57 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:57.521+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:57.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:57 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:57.521+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:57.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:57 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:57.521+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:57.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:57 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:57.521+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:28:58.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:57 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:57.836+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:58.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:57 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:57.836+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:58.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:57 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:57.836+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:58.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:57 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:57.836+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.859245+0000 front 2024-04-26T20:28:32.859347+0000 (oldest deadline 2024-04-26T20:28:55.159399+0000) 2024-04-26T20:28:58.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:57 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:57.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:58.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:57 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:57.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:58.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:57 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:57.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:58.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:57 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:57.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:58.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:57 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:57.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:28:58.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:58 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:58.364+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:58.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:58 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:58.364+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:58.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:58 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:58.364+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:58.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:58 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:58.364+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:58.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:58 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:58.364+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:28:58.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:58 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:58.629+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:58.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:58 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:58.629+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:58.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:58 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:58.629+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:58.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:58 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:58.629+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:28:58.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:58 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:58.551+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:58.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:58 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:58.551+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:58.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:58 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:58.551+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:58.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:58 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:58.551+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:28:59.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:58 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:58.833+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:04.656075+0000 front 2024-04-26T20:28:04.656049+0000 (oldest deadline 2024-04-26T20:28:29.956178+0000) 2024-04-26T20:28:59.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:58 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:58.833+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:13.457271+0000 front 2024-04-26T20:28:13.457426+0000 (oldest deadline 2024-04-26T20:28:38.757329+0000) 2024-04-26T20:28:59.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:58 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:58.833+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:24.658700+0000 front 2024-04-26T20:28:24.658485+0000 (oldest deadline 2024-04-26T20:28:46.958316+0000) 2024-04-26T20:28:59.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:58 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7[50535]: 2024-04-26T20:28:58.833+0000 7f1b28765700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.859245+0000 front 2024-04-26T20:28:32.859347+0000 (oldest deadline 2024-04-26T20:28:55.159399+0000) 2024-04-26T20:28:59.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:58 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:58.737+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:28:59.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:58 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:58.737+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:28:59.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:58 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:58.737+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:28:59.036 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:58 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:58.737+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:28:59.036 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:58 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:58.737+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:28:59.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:59 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:59.404+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:28:59.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:59 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:59.404+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:28:59.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:59 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:59.404+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:28:59.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:59 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:59.404+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:28:59.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:28:59 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:28:59.404+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:28:59.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:59 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:59.672+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:28:59.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:59 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:59.672+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:28:59.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:59 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:59.672+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:28:59.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:28:59 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:28:59.672+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:28:59.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:59 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:59.561+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:28:59.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:59 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:59.561+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:28:59.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:59 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:59.561+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:28:59.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:28:59 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:28:59.561+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:00.034 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:59 smithi103 podman[68481]: 2024-04-26 20:28:59.817344179 +0000 UTC m=+5.558552530 container stop 07883f983056b0cfe668da11462cb220f591345bdbb95e5eb120a6162b93b853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T20:29:00.035 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:28:59 smithi103 podman[68481]: 2024-04-26 20:28:59.818026856 +0000 UTC m=+5.559235219 container died 07883f983056b0cfe668da11462cb220f591345bdbb95e5eb120a6162b93b853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T20:29:00.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:59 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:59.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:00.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:59 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:59.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:00.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:59 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:59.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:00.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:59 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:59.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:00.036 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:28:59 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:28:59.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:00.414 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:29:00 smithi103 podman[68481]: 2024-04-26 20:29:00.196291544 +0000 UTC m=+5.937499894 container cleanup 07883f983056b0cfe668da11462cb220f591345bdbb95e5eb120a6162b93b853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T20:29:00.414 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:29:00 smithi103 bash[68481]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7 2024-04-26T20:29:00.722 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:29:00 smithi103 podman[68509]: 2024-04-26 20:29:00.414107869 +0000 UTC m=+0.588649971 container remove 07883f983056b0cfe668da11462cb220f591345bdbb95e5eb120a6162b93b853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:29:00.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:00 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:00.359+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:00.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:00 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:00.359+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:00.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:00 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:00.359+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:00.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:00 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:00.359+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:00.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:00 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:00.359+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:00.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:00 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:00.706+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:29:00.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:00 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:00.706+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:29:00.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:00 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:00.706+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:29:00.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:00 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:00.706+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:29:00.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:00 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:00.554+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:00.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:00 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:00.554+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:00.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:00 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:00.554+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:00.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:00 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:00.554+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:00.996 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:29:00 smithi103 podman[68599]: 2024-04-26 20:29:00.756062608 +0000 UTC m=+0.018410764 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:29:00.997 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:29:00 smithi103 podman[68599]: 2024-04-26 20:29:00.873469488 +0000 UTC m=+0.135817673 container create bdd86aa023f64ee32c1373bc2ddbaf994afc1333fe1fa42a06ba1e26e5316a7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:29:00.997 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:00 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:00.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:00.997 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:00 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:00.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:00.998 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:00 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:00.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:00.998 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:00 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:00.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:00.998 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:00 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:00.722+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:01.284 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:29:01 smithi103 podman[68599]: 2024-04-26 20:29:01.109678858 +0000 UTC m=+0.372027002 container init bdd86aa023f64ee32c1373bc2ddbaf994afc1333fe1fa42a06ba1e26e5316a7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:29:01.284 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:29:01 smithi103 podman[68599]: 2024-04-26 20:29:01.112728647 +0000 UTC m=+0.375076799 container start bdd86aa023f64ee32c1373bc2ddbaf994afc1333fe1fa42a06ba1e26e5316a7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:29:01.284 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:29:01 smithi103 podman[68599]: 2024-04-26 20:29:01.201309563 +0000 UTC m=+0.463657712 container attach bdd86aa023f64ee32c1373bc2ddbaf994afc1333fe1fa42a06ba1e26e5316a7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T20:29:01.709 INFO:journalctl@ceph.osd.7.smithi103.stdout:Apr 26 20:29:01 smithi103 podman[68599]: 2024-04-26 20:29:01.402936653 +0000 UTC m=+0.665284791 container died bdd86aa023f64ee32c1373bc2ddbaf994afc1333fe1fa42a06ba1e26e5316a7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T20:29:01.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:01 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:01.400+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:01.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:01 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:01.400+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:01.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:01 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:01.400+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:01.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:01 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:01.400+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:01.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:01 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:01.400+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:01.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:01 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:01.741+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:29:01.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:01 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:01.741+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:29:01.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:01 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:01.741+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:29:01.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:01 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:01.741+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:29:01.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:01 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:01.547+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:01.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:01 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:01.547+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:01.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:01 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:01.547+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:01.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:01 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:01.547+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:02.005 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:01 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:01.708+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:02.005 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:01 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:01.708+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:02.005 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:01 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:01.708+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:02.006 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:01 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:01.708+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:02.006 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:01 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:01.708+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:02.024 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T20:29:02.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:02.064 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T20:29:02.064 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-26T20:29:02.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.8 2024-04-26T20:29:02.370 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:02 smithi204 systemd[1]: Stopping Ceph osd.8 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:02.631 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:02.369+0000 7f5adf881700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:02.631 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:02.369+0000 7f5adf881700 -1 osd.8 100 *** Got signal Terminated *** 2024-04-26T20:29:02.632 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:02.369+0000 7f5adf881700 -1 osd.8 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:29:02.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:02.560+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:02.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:02.560+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:02.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:02.560+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:02.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:02.560+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:02.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:02.426+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:02.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:02.426+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:02.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:02.426+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:02.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:02.426+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:02.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:02 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:02.426+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:03.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:02 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:02.684+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:03.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:02 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:02.684+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:03.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:02 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:02.684+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:03.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:02 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:02.684+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:03.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:02 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:02.684+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:03.131 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:02.756+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:29:03.131 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:02.756+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:29:03.132 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:02.756+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:29:03.132 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:02 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:02.756+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:29:03.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:03 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:03.469+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:03.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:03 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:03.469+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:03.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:03 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:03.469+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:03.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:03 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:03.469+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:03.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:03 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:03.469+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:03.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:03 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:03.603+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:03.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:03 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:03.603+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:03.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:03 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:03.603+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:03.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:03 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:03.603+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:03.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:03 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:03.781+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:29:03.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:03 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:03.781+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:29:03.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:03 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:03.781+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:29:03.883 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:03 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:03.781+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:29:03.883 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:03 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:03.781+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.142042+0000 front 2024-04-26T20:28:41.142067+0000 (oldest deadline 2024-04-26T20:29:02.842253+0000) 2024-04-26T20:29:04.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:03 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:03.667+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:04.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:03 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:03.667+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:04.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:03 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:03.667+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:04.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:03 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:03.667+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:04.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:03 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:03.667+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:04.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:04 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:04.432+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:04.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:04 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:04.432+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:04.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:04 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:04.432+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:04.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:04 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:04.432+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:04.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:04 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:04.432+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:04.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:04.570+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:04.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:04.570+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:04.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:04.570+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:04.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:04.570+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:04.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:04.804+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:29:04.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:04.804+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:29:04.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:04.804+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:29:04.883 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:04.804+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:29:04.883 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:04.804+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.142042+0000 front 2024-04-26T20:28:41.142067+0000 (oldest deadline 2024-04-26T20:29:02.842253+0000) 2024-04-26T20:29:05.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:04 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:04.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:05.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:04 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:04.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:05.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:04 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:04.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:05.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:04 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:04.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:05.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:04 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:04.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:05.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:04 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:04.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:05.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:05 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:05.443+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:05.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:05 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:05.443+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:05.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:05 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:05.443+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:05.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:05 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:05.443+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:05.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:05 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:05.443+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:05.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:05 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:05.443+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:05.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:05 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:05.529+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:05.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:05 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:05.529+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:05.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:05 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:05.529+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:05.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:05 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:05.529+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:05.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:05 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:05.773+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:29:05.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:05 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:05.773+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:29:05.883 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:05 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:05.773+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:29:05.883 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:05 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:05.773+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:29:05.883 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:05 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:05.773+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.142042+0000 front 2024-04-26T20:28:41.142067+0000 (oldest deadline 2024-04-26T20:29:02.842253+0000) 2024-04-26T20:29:06.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:05 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:05.635+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:06.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:05 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:05.635+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:06.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:05 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:05.635+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:06.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:05 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:05.635+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:06.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:05 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:05.635+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:06.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:05 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:05.635+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:06.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:06.401+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:06.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:06.401+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:06.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:06.401+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:06.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:06.401+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:06.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:06.401+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:06.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:06 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:06.401+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:06.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:06.777+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.836039+0000 front 2024-04-26T20:28:06.836059+0000 (oldest deadline 2024-04-26T20:28:30.336064+0000) 2024-04-26T20:29:06.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:06.777+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.937685+0000 front 2024-04-26T20:28:14.937782+0000 (oldest deadline 2024-04-26T20:28:40.237884+0000) 2024-04-26T20:29:06.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:06.777+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:23.039754+0000 front 2024-04-26T20:28:23.039903+0000 (oldest deadline 2024-04-26T20:28:47.739561+0000) 2024-04-26T20:29:06.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:06.777+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.640554+0000 front 2024-04-26T20:28:30.640541+0000 (oldest deadline 2024-04-26T20:28:55.940484+0000) 2024-04-26T20:29:06.882 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8[51142]: 2024-04-26T20:29:06.777+0000 7f5ad7e6d700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.142042+0000 front 2024-04-26T20:28:41.142067+0000 (oldest deadline 2024-04-26T20:29:02.842253+0000) 2024-04-26T20:29:06.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:06.565+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:06.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:06.565+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:06.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:06.565+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:06.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:06.565+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:06.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:06 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:06.565+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:07.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:06 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:06.622+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:07.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:06 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:06.622+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:07.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:06 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:06.622+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:07.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:06 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:06.622+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:07.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:06 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:06.622+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:07.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:06 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:06.622+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:07.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:07 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:07.381+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:07.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:07 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:07.381+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:07.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:07 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:07.381+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:07.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:07 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:07.381+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:07.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:07 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:07.381+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:07.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:07 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:07.381+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:07.881 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:07 smithi204 podman[68809]: 2024-04-26 20:29:07.712811031 +0000 UTC m=+5.463763391 container died 226eaa96b768a500f60dad0e9b4b6480ca6afa1e473decdf76dc9aed55c0fdc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T20:29:07.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:07 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:07.535+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:07.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:07 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:07.535+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:07.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:07 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:07.535+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:07.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:07 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:07.535+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:07.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:07 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:07.535+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:08.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:07 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:07.621+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:08.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:07 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:07.621+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:08.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:07 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:07.621+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:08.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:07 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:07.621+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:08.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:07 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:07.621+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:08.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:07 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:07.621+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:08.338 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:08 smithi204 podman[68809]: 2024-04-26 20:29:08.086838317 +0000 UTC m=+5.837790683 container cleanup 226eaa96b768a500f60dad0e9b4b6480ca6afa1e473decdf76dc9aed55c0fdc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:29:08.338 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:08 smithi204 bash[68809]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8 2024-04-26T20:29:08.631 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:08 smithi204 podman[68826]: 2024-04-26 20:29:08.338201461 +0000 UTC m=+0.619226045 container remove 226eaa96b768a500f60dad0e9b4b6480ca6afa1e473decdf76dc9aed55c0fdc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:29:08.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:08 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:08.519+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:08.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:08 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:08.519+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:08.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:08 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:08.519+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:08.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:08 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:08.519+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:08.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:08 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:08.519+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:08.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:08 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:08.354+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:08.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:08 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:08.354+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:08.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:08 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:08.354+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:08.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:08 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:08.354+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:08.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:08 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:08.354+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:08.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:08 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:08.354+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:08.908 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:08 smithi204 podman[68914]: 2024-04-26 20:29:08.560589923 +0000 UTC m=+0.017890429 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:29:08.909 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:08 smithi204 podman[68914]: 2024-04-26 20:29:08.664331417 +0000 UTC m=+0.121631901 container create 665a7a3743a93f30a19becb11cd279041aec50ef27d8ed67ee1748ac612c6b0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:29:09.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:08 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:08.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:09.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:08 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:08.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:09.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:08 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:08.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:09.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:08 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:08.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:09.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:08 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:08.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:09.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:08 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:08.636+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:09.211 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:08 smithi204 podman[68914]: 2024-04-26 20:29:08.90856275 +0000 UTC m=+0.365863234 container init 665a7a3743a93f30a19becb11cd279041aec50ef27d8ed67ee1748ac612c6b0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T20:29:09.211 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:08 smithi204 podman[68914]: 2024-04-26 20:29:08.911896719 +0000 UTC m=+0.369197206 container start 665a7a3743a93f30a19becb11cd279041aec50ef27d8ed67ee1748ac612c6b0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS) 2024-04-26T20:29:09.211 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:09 smithi204 podman[68914]: 2024-04-26 20:29:09.017276052 +0000 UTC m=+0.474576543 container attach 665a7a3743a93f30a19becb11cd279041aec50ef27d8ed67ee1748ac612c6b0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-04-26T20:29:09.477 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 26 20:29:09 smithi204 podman[68914]: 2024-04-26 20:29:09.211251521 +0000 UTC m=+0.668552015 container died 665a7a3743a93f30a19becb11cd279041aec50ef27d8ed67ee1748ac612c6b0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T20:29:09.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:09.398+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:09.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:09.398+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:09.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:09.398+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:09.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:09.398+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:09.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:09.398+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:09.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:09 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:09.398+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:09.747 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:09 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:09.477+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:09.747 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:09 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:09.477+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:09.747 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:09 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:09.477+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:09.747 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:09 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:09.477+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:09.747 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:09 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:09.477+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:09.761 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.8.service' 2024-04-26T20:29:09.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:09.798 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-26T20:29:09.799 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-26T20:29:09.799 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.9 2024-04-26T20:29:10.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:09 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:09.615+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:10.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:09 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:09.615+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:10.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:09 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:09.615+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:10.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:09 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:09.615+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:10.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:09 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:09.615+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:10.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:09 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:09.615+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:10.237 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:09 smithi094 systemd[1]: Stopping Ceph osd.9 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:10.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:10.409+0000 7fa929200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:10.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:10.409+0000 7fa929200700 -1 osd.9 100 *** Got signal Terminated *** 2024-04-26T20:29:10.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:10.409+0000 7fa929200700 -1 osd.9 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:29:10.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:10.419+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:10.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:10.419+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:10.739 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:10.419+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:10.739 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:10.419+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:10.739 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:10.419+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:10.739 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:10 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:10.419+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:10.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:10 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:10.462+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:10.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:10 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:10.462+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:10.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:10 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:10.462+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:10.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:10 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:10.462+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:10.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:10 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:10.462+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:11.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:10 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:10.587+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:11.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:10 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:10.587+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:11.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:10 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:10.587+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:11.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:10 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:10.587+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:11.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:10 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:10.587+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:11.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:10 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:10.587+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:11.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:11 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:11.411+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:11.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:11 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:11.411+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:11.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:11 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:11.411+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:11.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:11 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:11.411+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:11.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:11 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:11.411+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:11.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:11 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:11.411+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:11.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:11 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:11.416+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:11.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:11 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:11.416+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:11.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:11 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:11.416+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:11.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:11 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:11.416+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:11.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:11 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:11.416+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:12.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:11 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:11.547+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:12.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:11 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:11.547+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:12.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:11 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:11.547+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:12.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:11 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:11.547+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:12.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:11 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:11.547+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:12.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:11 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:11.547+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:12.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:12 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:12.415+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:12.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:12 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:12.415+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:12.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:12 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:12.415+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:12.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:12 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:12.415+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:12.739 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:12 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:12.415+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:12.739 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:12 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:12.415+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:12.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:12 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:12.393+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:12.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:12 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:12.393+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:12.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:12 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:12.393+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:12.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:12 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:12.393+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:12.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:12 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:12.393+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:13.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:12 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:12.577+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:13.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:12 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:12.577+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:13.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:12 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:12.577+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:13.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:12 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:12.577+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:13.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:12 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:12.577+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:13.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:12 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:12.577+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:13.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:13 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:13.351+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:13.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:13 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:13.351+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:13.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:13 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:13.351+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:13.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:13 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:13.351+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:13.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:13 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:13.351+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:13.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:13 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:13.431+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:13.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:13 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:13.431+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:13.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:13 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:13.431+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:13.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:13 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:13.431+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:13.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:13 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:13.431+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:13.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:13 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:13.431+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:14.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:13 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:13.595+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:14.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:13 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:13.595+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:14.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:13 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:13.595+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:14.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:13 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:13.595+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:14.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:13 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:13.595+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:14.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:13 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:13.595+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:14.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:13 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:13.595+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:14.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:14 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:14.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:14.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:14 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:14.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:14.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:14 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:14.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:14.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:14 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:14.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:14.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:14 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:14.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:14.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:14 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:14.418+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:14.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:14 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:14.418+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:14.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:14 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:14.418+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:14.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:14 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:14.418+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:14.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:14 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:14.418+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:14.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:14 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:14.418+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:15.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:14 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:14.625+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:15.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:14 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:14.625+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:15.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:14 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:14.625+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:15.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:14 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:14.625+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:15.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:14 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:14.625+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:15.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:14 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:14.625+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:15.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:14 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:14.625+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:15.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:15 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:15.379+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:58.217708+0000 front 2024-04-26T20:27:58.217358+0000 (oldest deadline 2024-04-26T20:28:23.417881+0000) 2024-04-26T20:29:15.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:15 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:15.379+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:06.821262+0000 front 2024-04-26T20:28:06.821323+0000 (oldest deadline 2024-04-26T20:28:30.222061+0000) 2024-04-26T20:29:15.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:15 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:15.379+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.825056+0000 front 2024-04-26T20:28:14.825300+0000 (oldest deadline 2024-04-26T20:28:40.125339+0000) 2024-04-26T20:29:15.737 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:15 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:15.379+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.827507+0000 front 2024-04-26T20:28:21.827980+0000 (oldest deadline 2024-04-26T20:28:45.927402+0000) 2024-04-26T20:29:15.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:15 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:15.379+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:32.329785+0000 front 2024-04-26T20:28:32.329912+0000 (oldest deadline 2024-04-26T20:28:58.229992+0000) 2024-04-26T20:29:15.738 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:15 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9[59959]: 2024-04-26T20:29:15.379+0000 7fa91fa00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:41.731793+0000 front 2024-04-26T20:28:41.732040+0000 (oldest deadline 2024-04-26T20:29:04.631901+0000) 2024-04-26T20:29:15.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:15 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:15.389+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:15.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:15 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:15.389+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:15.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:15 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:15.389+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:15.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:15 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:15.389+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:15.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:15 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:15.389+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:16.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:15 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:15.670+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:16.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:15 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:15.670+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:16.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:15 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:15.670+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:16.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:15 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:15.670+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:16.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:15 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:15.670+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:16.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:15 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:15.670+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:16.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:15 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:15.670+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:16.283 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:16 smithi094 podman[83762]: 2024-04-26 20:29:15.999991728 +0000 UTC m=+5.728347031 container died 9f24579bc988f7448fcba8aa5d6ab5e2ee8f439f2b69f26d7b40e82b431e13ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T20:29:16.613 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:16 smithi094 podman[83762]: 2024-04-26 20:29:16.377466899 +0000 UTC m=+6.105822231 container cleanup 9f24579bc988f7448fcba8aa5d6ab5e2ee8f439f2b69f26d7b40e82b431e13ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T20:29:16.613 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:16 smithi094 bash[83762]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9 2024-04-26T20:29:16.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:16 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:16.368+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:16.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:16 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:16.368+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:16.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:16 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:16.368+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:16.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:16 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:16.368+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:16.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:16 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:16.368+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:16.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:16 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:16.368+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:16.987 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:16 smithi094 podman[83780]: 2024-04-26 20:29:16.611987285 +0000 UTC m=+0.599203356 container remove 9f24579bc988f7448fcba8aa5d6ab5e2ee8f439f2b69f26d7b40e82b431e13ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T20:29:17.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:16 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:16.705+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:17.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:16 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:16.705+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:17.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:16 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:16.705+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:17.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:16 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:16.705+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:17.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:16 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:16.705+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:17.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:16 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:16.705+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:17.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:16 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:16.705+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:17.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:17 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:17.340+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:17.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:17 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:17.340+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:17.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:17 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:17.340+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:17.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:17 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:17.340+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:17.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:17 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:17.340+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:17.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:17 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:17.340+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:17.701 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:17 smithi094 podman[83868]: 2024-04-26 20:29:17.312615045 +0000 UTC m=+0.035315193 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:29:17.701 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:17 smithi094 podman[83868]: 2024-04-26 20:29:17.413033849 +0000 UTC m=+0.135734034 container create e277cd0f71893066037b306234163d651efa4e9b825bd9673f5ca8913fd88264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:29:17.987 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:17 smithi094 podman[83868]: 2024-04-26 20:29:17.70037356 +0000 UTC m=+0.423073764 container init e277cd0f71893066037b306234163d651efa4e9b825bd9673f5ca8913fd88264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:29:17.988 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:17 smithi094 podman[83868]: 2024-04-26 20:29:17.713129711 +0000 UTC m=+0.435829868 container start e277cd0f71893066037b306234163d651efa4e9b825bd9673f5ca8913fd88264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T20:29:17.988 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:17 smithi094 podman[83868]: 2024-04-26 20:29:17.799089824 +0000 UTC m=+0.521790062 container attach e277cd0f71893066037b306234163d651efa4e9b825bd9673f5ca8913fd88264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T20:29:18.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:17 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:17.729+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:18.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:17 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:17.729+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:18.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:17 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:17.729+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:18.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:17 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:17.729+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:18.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:17 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:17.729+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:18.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:17 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:17.729+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:18.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:17 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:17.729+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:18.487 INFO:journalctl@ceph.osd.9.smithi094.stdout:Apr 26 20:29:18 smithi094 podman[83868]: 2024-04-26 20:29:18.17239757 +0000 UTC m=+0.895097758 container died e277cd0f71893066037b306234163d651efa4e9b825bd9673f5ca8913fd88264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-26T20:29:18.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:18 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:18.363+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:18.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:18 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:18.363+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:18.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:18 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:18.363+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:18.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:18 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:18.363+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:18.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:18 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:18.363+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:18.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:18 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:18.363+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:18.902 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.9.service' 2024-04-26T20:29:18.998 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:18.998 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-26T20:29:18.998 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-26T20:29:18.999 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.10 2024-04-26T20:29:19.024 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:18 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:18.714+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:19.024 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:18 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:18.714+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:19.024 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:18 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:18.714+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:19.024 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:18 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:18.714+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:19.024 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:18 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:18.714+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:19.025 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:18 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:18.714+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:19.025 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:18 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:18.714+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:19.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 systemd[1]: Stopping Ceph osd.10 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:19.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:19 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:19.371+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:19.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:19 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:19.371+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:19.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:19 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:19.371+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:19.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:19 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:19.371+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:19.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:19 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:19.371+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:19.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:19 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:19.371+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:19.757 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.290+0000 7f9dea0c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:19.757 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.290+0000 7f9dea0c3700 -1 osd.10 100 *** Got signal Terminated *** 2024-04-26T20:29:19.758 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.290+0000 7f9dea0c3700 -1 osd.10 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:29:19.758 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:19.758 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:19.758 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:19.758 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:19.758 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:19.758 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:19.759 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:19 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:19.756+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:20.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:20 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:20.334+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:20.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:20 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:20.334+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:20.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:20 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:20.334+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:20.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:20 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:20.334+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:20.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:20 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:20.334+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:20.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:20 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:20.334+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:20.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:20 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:20.334+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:21.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:20 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:20.747+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:21.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:20 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:20.747+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:21.037 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:20 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:20.747+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:21.037 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:20 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:20.747+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:21.038 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:20 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:20.747+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:21.038 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:20 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:20.747+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:21.038 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:20 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:20.747+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:21.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:21 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:21.330+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:21.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:21 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:21.330+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:21.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:21 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:21.330+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:21.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:21 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:21.330+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:21.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:21 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:21.330+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:21.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:21 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:21.330+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:21.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:21 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:21.330+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:22.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:21 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:21.763+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:22.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:21 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:21.763+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:22.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:21 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:21.763+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:22.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:21 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:21.763+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:22.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:21 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:21.763+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:22.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:21 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:21.763+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:22.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:21 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:21.763+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:22.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:22 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:22.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:22.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:22 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:22.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:22.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:22 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:22.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:22.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:22 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:22.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:22.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:22 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:22.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:22.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:22 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:22.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:22.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:22 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:22.350+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:23.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:22 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:22.801+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:23.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:22 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:22.801+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:23.285 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:22 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:22.801+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:23.285 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:22 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:22.801+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:23.285 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:22 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:22.801+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:23.285 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:22 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:22.801+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:23.285 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:22 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:22.801+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:23.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:23 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:23.342+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:23.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:23 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:23.342+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:23.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:23 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:23.342+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:23.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:23 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:23.342+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:23.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:23 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:23.342+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:23.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:23 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:23.342+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:23.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:23 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:23.342+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:24.034 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:23 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:23.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:56.960811+0000 front 2024-04-26T20:27:56.960678+0000 (oldest deadline 2024-04-26T20:28:20.460649+0000) 2024-04-26T20:29:24.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:23 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:23.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.962785+0000 front 2024-04-26T20:28:07.962818+0000 (oldest deadline 2024-04-26T20:28:29.662942+0000) 2024-04-26T20:29:24.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:23 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:23.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:16.064427+0000 front 2024-04-26T20:28:16.064523+0000 (oldest deadline 2024-04-26T20:28:38.364363+0000) 2024-04-26T20:29:24.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:23 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:23.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6814 osd.3 since back 2024-04-26T20:28:21.865717+0000 front 2024-04-26T20:28:21.865826+0000 (oldest deadline 2024-04-26T20:28:45.965582+0000) 2024-04-26T20:29:24.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:23 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:23.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:30.666510+0000 front 2024-04-26T20:28:30.666537+0000 (oldest deadline 2024-04-26T20:28:55.366557+0000) 2024-04-26T20:29:24.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:23 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:23.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.567826+0000 front 2024-04-26T20:28:40.567807+0000 (oldest deadline 2024-04-26T20:29:04.067754+0000) 2024-04-26T20:29:24.035 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:23 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10[57551]: 2024-04-26T20:29:23.775+0000 7f9de66b7700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:49.269680+0000 front 2024-04-26T20:28:49.269566+0000 (oldest deadline 2024-04-26T20:29:12.769331+0000) 2024-04-26T20:29:24.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:24.374+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:24.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:24.374+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:24.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:24.374+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:24.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:24.374+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:24.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:24.374+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:24.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:24.374+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:24.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:24 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:24.374+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:24.916 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:24 smithi103 podman[68704]: 2024-04-26 20:29:24.647845434 +0000 UTC m=+5.456885031 container died b51b62f432c399f866be67d13f923ff68a5f8a42ca70bef1445cfc13d4c3b502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T20:29:25.257 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:25 smithi103 podman[68704]: 2024-04-26 20:29:25.006688884 +0000 UTC m=+5.815728477 container cleanup b51b62f432c399f866be67d13f923ff68a5f8a42ca70bef1445cfc13d4c3b502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T20:29:25.258 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:25 smithi103 bash[68704]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10 2024-04-26T20:29:25.534 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:25 smithi103 podman[68730]: 2024-04-26 20:29:25.257814051 +0000 UTC m=+0.610236502 container remove b51b62f432c399f866be67d13f923ff68a5f8a42ca70bef1445cfc13d4c3b502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T20:29:25.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:25 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:25.362+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:25.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:25 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:25.362+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:25.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:25 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:25.362+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:25.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:25 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:25.362+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:25.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:25 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:25.362+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:25.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:25 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:25.362+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:25.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:25 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:25.362+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:25.861 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:25 smithi103 podman[68820]: 2024-04-26 20:29:25.567003425 +0000 UTC m=+0.100720464 container create 25d336c5b5c9a52618144e42ccf841cba7168d3461f56584a1c55245bd5815ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T20:29:25.861 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:25 smithi103 podman[68820]: 2024-04-26 20:29:25.484177814 +0000 UTC m=+0.017894882 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:29:25.861 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:25 smithi103 podman[68820]: 2024-04-26 20:29:25.80311435 +0000 UTC m=+0.336831391 container init 25d336c5b5c9a52618144e42ccf841cba7168d3461f56584a1c55245bd5815ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T20:29:25.861 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:25 smithi103 podman[68820]: 2024-04-26 20:29:25.806434108 +0000 UTC m=+0.340151148 container start 25d336c5b5c9a52618144e42ccf841cba7168d3461f56584a1c55245bd5815ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:29:25.861 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:25 smithi103 podman[68820]: 2024-04-26 20:29:25.860735825 +0000 UTC m=+0.394452882 container attach 25d336c5b5c9a52618144e42ccf841cba7168d3461f56584a1c55245bd5815ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T20:29:26.284 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:26 smithi103 podman[68820]: 2024-04-26 20:29:26.09207523 +0000 UTC m=+0.625792280 container died 25d336c5b5c9a52618144e42ccf841cba7168d3461f56584a1c55245bd5815ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T20:29:26.607 INFO:journalctl@ceph.osd.10.smithi103.stdout:Apr 26 20:29:26 smithi103 podman[68846]: 2024-04-26 20:29:26.589207107 +0000 UTC m=+0.488149423 container remove 25d336c5b5c9a52618144e42ccf841cba7168d3461f56584a1c55245bd5815ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-26T20:29:26.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.10.service' 2024-04-26T20:29:26.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:26 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.353+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:26.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:26 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.353+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:26.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:26 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.353+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:26.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:26 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.353+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:26.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:26 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.353+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:26.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:26 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.353+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:26.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:26 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.353+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:26.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:26.692 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-26T20:29:26.693 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-26T20:29:26.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.11 2024-04-26T20:29:27.000 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:26 smithi204 systemd[1]: Stopping Ceph osd.11 for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:27.346 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.999+0000 7feee5c79700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:27.347 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.999+0000 7feee5c79700 -1 osd.11 100 *** Got signal Terminated *** 2024-04-26T20:29:27.347 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:26.999+0000 7feee5c79700 -1 osd.11 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:29:27.347 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:27.345+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:27.347 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:27.345+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:27.347 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:27.345+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:27.347 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:27.345+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:27.347 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:27.345+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:27.348 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:27.345+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:27.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:27 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:27.345+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:28.631 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:28 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:28.378+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:28.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:28 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:28.378+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:28.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:28 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:28.378+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:28.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:28 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:28.378+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:28.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:28 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:28.378+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:28.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:28 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:28.378+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:28.632 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:28 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:28.378+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:29.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:29.411+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:29.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:29.411+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:29.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:29.411+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:29.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:29.411+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:29.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:29.411+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:29.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:29.411+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:29.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:29 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:29.411+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:30.783 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:30.443+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:30.783 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:30.443+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:30.783 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:30.443+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:30.783 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:30.443+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:30.783 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:30.443+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:30.783 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:30.443+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:30.783 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:30.443+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:30.784 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:30 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:30.443+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6820 osd.8 since back 2024-04-26T20:29:06.066196+0000 front 2024-04-26T20:29:06.066231+0000 (oldest deadline 2024-04-26T20:29:30.166300+0000) 2024-04-26T20:29:31.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:31.474+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6806 osd.0 since back 2024-04-26T20:27:59.352696+0000 front 2024-04-26T20:27:59.352925+0000 (oldest deadline 2024-04-26T20:28:22.852630+0000) 2024-04-26T20:29:31.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:31.475+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-04-26T20:28:07.553894+0000 front 2024-04-26T20:28:07.553872+0000 (oldest deadline 2024-04-26T20:28:29.853878+0000) 2024-04-26T20:29:31.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:31.475+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-26T20:28:14.455312+0000 front 2024-04-26T20:28:14.455388+0000 (oldest deadline 2024-04-26T20:28:37.955381+0000) 2024-04-26T20:29:31.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:31.475+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-04-26T20:28:33.060967+0000 front 2024-04-26T20:28:33.060977+0000 (oldest deadline 2024-04-26T20:28:55.360881+0000) 2024-04-26T20:29:31.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:31.475+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-26T20:28:40.562274+0000 front 2024-04-26T20:28:40.562337+0000 (oldest deadline 2024-04-26T20:29:05.862291+0000) 2024-04-26T20:29:31.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:31.475+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.94:6822 osd.6 since back 2024-04-26T20:28:50.364425+0000 front 2024-04-26T20:28:50.364389+0000 (oldest deadline 2024-04-26T20:29:16.264067+0000) 2024-04-26T20:29:31.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:31.475+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-04-26T20:28:59.065348+0000 front 2024-04-26T20:28:59.065385+0000 (oldest deadline 2024-04-26T20:29:20.165293+0000) 2024-04-26T20:29:31.883 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:31 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11[58200]: 2024-04-26T20:29:31.475+0000 7feede265700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.204:6820 osd.8 since back 2024-04-26T20:29:06.066196+0000 front 2024-04-26T20:29:06.066231+0000 (oldest deadline 2024-04-26T20:29:30.166300+0000) 2024-04-26T20:29:32.609 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:32 smithi204 podman[69019]: 2024-04-26 20:29:32.356459217 +0000 UTC m=+5.468695162 container died 8eae1083e651aeba0eab9ba25678a861ec8ad22009562066a31d4e01ac120214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T20:29:32.881 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:32 smithi204 podman[69019]: 2024-04-26 20:29:32.699400548 +0000 UTC m=+5.811636491 container cleanup 8eae1083e651aeba0eab9ba25678a861ec8ad22009562066a31d4e01ac120214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-26T20:29:32.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:32 smithi204 bash[69019]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11 2024-04-26T20:29:33.301 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:32 smithi204 podman[69037]: 2024-04-26 20:29:32.91738123 +0000 UTC m=+0.556161847 container remove 8eae1083e651aeba0eab9ba25678a861ec8ad22009562066a31d4e01ac120214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T20:29:33.587 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:33 smithi204 podman[69125]: 2024-04-26 20:29:33.301397633 +0000 UTC m=+0.101887081 container create 91f97841335adffed154faebc4c122fe3b475cf0027b675550e392e5ca939f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T20:29:33.587 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:33 smithi204 podman[69125]: 2024-04-26 20:29:33.217656299 +0000 UTC m=+0.018145763 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:29:33.587 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:33 smithi204 podman[69125]: 2024-04-26 20:29:33.521569991 +0000 UTC m=+0.322059439 container init 91f97841335adffed154faebc4c122fe3b475cf0027b675550e392e5ca939f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-26T20:29:33.588 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:33 smithi204 podman[69125]: 2024-04-26 20:29:33.52507023 +0000 UTC m=+0.325559686 container start 91f97841335adffed154faebc4c122fe3b475cf0027b675550e392e5ca939f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-26T20:29:33.588 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:33 smithi204 podman[69125]: 2024-04-26 20:29:33.586993087 +0000 UTC m=+0.387482545 container attach 91f97841335adffed154faebc4c122fe3b475cf0027b675550e392e5ca939f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-26T20:29:33.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:33 smithi204 podman[69125]: 2024-04-26 20:29:33.812298234 +0000 UTC m=+0.612787685 container died 91f97841335adffed154faebc4c122fe3b475cf0027b675550e392e5ca939f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T20:29:34.323 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:34 smithi204 podman[69151]: 2024-04-26 20:29:34.298807224 +0000 UTC m=+0.478768312 container remove 91f97841335adffed154faebc4c122fe3b475cf0027b675550e392e5ca939f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:29:34.323 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 26 20:29:34 smithi204 systemd[1]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.11.service: Deactivated successfully. 2024-04-26T20:29:34.326 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@osd.11.service' 2024-04-26T20:29:34.362 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:34.362 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-26T20:29:34.363 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-26T20:29:34.363 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.a 2024-04-26T20:29:34.737 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:29:34 smithi094 systemd[1]: Stopping Ceph mds.a for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:35.237 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:29:34 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a[66827]: 2024-04-26T20:29:34.954+0000 7f64d4a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:35.237 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:29:34 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a[66827]: 2024-04-26T20:29:34.954+0000 7f64d4a00700 -1 mds.a *** got signal Terminated *** 2024-04-26T20:29:36.987 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:29:36 smithi094 podman[83997]: 2024-04-26 20:29:36.687914633 +0000 UTC m=+1.876769529 container died 9c1d2dbf8e40fa2d90085cf93c8b603b031b4801f3ff55d4f5176e98875bec1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T20:29:37.383 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:29:37 smithi094 podman[83997]: 2024-04-26 20:29:37.114000492 +0000 UTC m=+2.302855382 container cleanup 9c1d2dbf8e40fa2d90085cf93c8b603b031b4801f3ff55d4f5176e98875bec1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:29:37.383 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:29:37 smithi094 bash[83997]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a 2024-04-26T20:29:37.737 INFO:journalctl@ceph.mds.a.smithi094.stdout:Apr 26 20:29:37 smithi094 podman[84011]: 2024-04-26 20:29:37.38190633 +0000 UTC m=+0.678116104 container remove 9c1d2dbf8e40fa2d90085cf93c8b603b031b4801f3ff55d4f5176e98875bec1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-26T20:29:37.919 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.a.service' 2024-04-26T20:29:38.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:38.031 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-26T20:29:38.031 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-26T20:29:38.031 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.d 2024-04-26T20:29:38.487 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:29:38 smithi094 systemd[1]: Stopping Ceph mds.d for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:38.987 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:29:38 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d[65899]: 2024-04-26T20:29:38.669+0000 7f4972200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:38.987 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:29:38 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d[65899]: 2024-04-26T20:29:38.669+0000 7f4972200700 -1 mds.d *** got signal Terminated *** 2024-04-26T20:29:40.282 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:29:40 smithi094 podman[84117]: 2024-04-26 20:29:40.020252526 +0000 UTC m=+1.507541588 container died 394cd22bae4c95b7bd8148e7cd903b03cc55c61d494cf779a0faa971aef101a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:29:40.637 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:29:40 smithi094 podman[84117]: 2024-04-26 20:29:40.393240372 +0000 UTC m=+1.880529466 container cleanup 394cd22bae4c95b7bd8148e7cd903b03cc55c61d494cf779a0faa971aef101a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T20:29:40.637 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:29:40 smithi094 bash[84117]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d 2024-04-26T20:29:40.987 INFO:journalctl@ceph.mds.d.smithi094.stdout:Apr 26 20:29:40 smithi094 podman[84131]: 2024-04-26 20:29:40.636075148 +0000 UTC m=+0.604349881 container remove 394cd22bae4c95b7bd8148e7cd903b03cc55c61d494cf779a0faa971aef101a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T20:29:41.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.d.service' 2024-04-26T20:29:41.212 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:41.212 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-26T20:29:41.213 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-26T20:29:41.213 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.g 2024-04-26T20:29:41.737 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:29:41 smithi094 systemd[1]: Stopping Ceph mds.g for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:42.237 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:29:41 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g[66477]: 2024-04-26T20:29:41.786+0000 7f0e5ac00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:42.237 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:29:41 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g[66477]: 2024-04-26T20:29:41.786+0000 7f0e5ac00700 -1 mds.g *** got signal Terminated *** 2024-04-26T20:29:43.088 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:29:42 smithi094 podman[84237]: 2024-04-26 20:29:42.787877721 +0000 UTC m=+1.137774484 container died 6bc36881c32d23d1322a2e537ac3d1aeaeb7ff2e80c428a0d211db7977aad451 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T20:29:43.488 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:29:43 smithi094 podman[84237]: 2024-04-26 20:29:43.205981947 +0000 UTC m=+1.555878710 container cleanup 6bc36881c32d23d1322a2e537ac3d1aeaeb7ff2e80c428a0d211db7977aad451 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T20:29:43.488 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:29:43 smithi094 bash[84237]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g 2024-04-26T20:29:43.960 INFO:journalctl@ceph.mds.g.smithi094.stdout:Apr 26 20:29:43 smithi094 podman[84251]: 2024-04-26 20:29:43.512273451 +0000 UTC m=+0.710809146 container remove 6bc36881c32d23d1322a2e537ac3d1aeaeb7ff2e80c428a0d211db7977aad451 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T20:29:43.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.g.service' 2024-04-26T20:29:44.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:44.092 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-26T20:29:44.092 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-26T20:29:44.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.j 2024-04-26T20:29:44.700 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:29:44 smithi094 systemd[1]: Stopping Ceph mds.j for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:44.989 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:29:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j[65528]: 2024-04-26T20:29:44.698+0000 7f6f15600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:44.989 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:29:44 smithi094 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j[65528]: 2024-04-26T20:29:44.698+0000 7f6f15600700 -1 mds.j *** got signal Terminated *** 2024-04-26T20:29:48.014 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:29:47 smithi094 podman[84358]: 2024-04-26 20:29:47.740016461 +0000 UTC m=+3.191383594 container died c8c81c34b8fb405424ea252609b5ee75e7996af673050ac08826ae48b5734a03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T20:29:48.377 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:29:48 smithi094 podman[84358]: 2024-04-26 20:29:48.134970893 +0000 UTC m=+3.586338039 container cleanup c8c81c34b8fb405424ea252609b5ee75e7996af673050ac08826ae48b5734a03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T20:29:48.377 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:29:48 smithi094 bash[84358]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j 2024-04-26T20:29:48.737 INFO:journalctl@ceph.mds.j.smithi094.stdout:Apr 26 20:29:48 smithi094 podman[84372]: 2024-04-26 20:29:48.376690595 +0000 UTC m=+0.616446909 container remove c8c81c34b8fb405424ea252609b5ee75e7996af673050ac08826ae48b5734a03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T20:29:48.870 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.j.service' 2024-04-26T20:29:48.977 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:48.977 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-26T20:29:48.977 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-26T20:29:48.978 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.b 2024-04-26T20:29:49.266 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:29:49 smithi103 systemd[1]: Stopping Ceph mds.b for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:49.534 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:29:49 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b[63233]: 2024-04-26T20:29:49.265+0000 7f2e0321a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:49.534 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:29:49 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b[63233]: 2024-04-26T20:29:49.265+0000 7f2e0321a700 -1 mds.b *** got signal Terminated *** 2024-04-26T20:29:52.883 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:29:52 smithi103 podman[68926]: 2024-04-26 20:29:52.626623587 +0000 UTC m=+3.454852507 container died f37d75c0bce39d78c82cdfd5baa0c86486540f8be7f4d03d64021a855a3e56fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:29:53.257 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:29:52 smithi103 podman[68926]: 2024-04-26 20:29:52.981427648 +0000 UTC m=+3.809656564 container cleanup f37d75c0bce39d78c82cdfd5baa0c86486540f8be7f4d03d64021a855a3e56fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T20:29:53.257 INFO:journalctl@ceph.mds.b.smithi103.stdout:Apr 26 20:29:52 smithi103 bash[68926]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-b 2024-04-26T20:29:53.402 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.b.service' 2024-04-26T20:29:53.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:53.437 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-26T20:29:53.437 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-26T20:29:53.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.e 2024-04-26T20:29:53.784 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:29:53 smithi103 systemd[1]: Stopping Ceph mds.e for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:53.784 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:29:53 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-e[62601]: 2024-04-26T20:29:53.734+0000 7fa7faf3d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:53.785 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:29:53 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-e[62601]: 2024-04-26T20:29:53.734+0000 7fa7faf3d700 -1 mds.e *** got signal Terminated *** 2024-04-26T20:29:55.185 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:29:54 smithi103 podman[69048]: 2024-04-26 20:29:54.913215499 +0000 UTC m=+1.265234701 container died e40f6bcc56a925a7975de3c301d5d8cd09fecd5371bd1eeaeb853e100f44ab57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-e, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T20:29:55.534 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:29:55 smithi103 podman[69048]: 2024-04-26 20:29:55.276671908 +0000 UTC m=+1.628691107 container cleanup e40f6bcc56a925a7975de3c301d5d8cd09fecd5371bd1eeaeb853e100f44ab57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:29:55.534 INFO:journalctl@ceph.mds.e.smithi103.stdout:Apr 26 20:29:55 smithi103 bash[69048]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-e 2024-04-26T20:29:55.694 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.e.service' 2024-04-26T20:29:55.726 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:29:55.726 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-26T20:29:55.726 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-26T20:29:55.726 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.h 2024-04-26T20:29:56.284 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:29:55 smithi103 systemd[1]: Stopping Ceph mds.h for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:29:56.284 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:29:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h[62919]: 2024-04-26T20:29:56.029+0000 7f5e55ff0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:29:56.284 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:29:56 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h[62919]: 2024-04-26T20:29:56.030+0000 7f5e55ff0700 -1 mds.h *** got signal Terminated *** 2024-04-26T20:30:00.290 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:30:00 smithi103 podman[69169]: 2024-04-26 20:30:00.037469505 +0000 UTC m=+4.103645348 container died 11411d6eab466099cf2335b2a5dd11a4afbeef392e52353b14dbacfdf6a504bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T20:30:00.664 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:30:00 smithi103 podman[69169]: 2024-04-26 20:30:00.388759705 +0000 UTC m=+4.454935544 container cleanup 11411d6eab466099cf2335b2a5dd11a4afbeef392e52353b14dbacfdf6a504bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T20:30:00.665 INFO:journalctl@ceph.mds.h.smithi103.stdout:Apr 26 20:30:00 smithi103 bash[69169]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-h 2024-04-26T20:30:00.816 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.h.service' 2024-04-26T20:30:00.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:30:00.849 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-26T20:30:00.849 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-26T20:30:00.850 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.k 2024-04-26T20:30:01.284 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:30:00 smithi103 systemd[1]: Stopping Ceph mds.k for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:30:01.285 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:30:01 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-k[62284]: 2024-04-26T20:30:01.159+0000 7f682e276700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:30:01.285 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:30:01 smithi103 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-k[62284]: 2024-04-26T20:30:01.159+0000 7f682e276700 -1 mds.k *** got signal Terminated *** 2024-04-26T20:30:04.018 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:30:03 smithi103 podman[69304]: 2024-04-26 20:30:03.722252531 +0000 UTC m=+2.663041491 container died 93948c78258d3dd6863350cb784d49c4271f115946ee8958d8afdb322dd0a10a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T20:30:04.284 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:30:04 smithi103 podman[69304]: 2024-04-26 20:30:04.108849154 +0000 UTC m=+3.049638125 container cleanup 93948c78258d3dd6863350cb784d49c4271f115946ee8958d8afdb322dd0a10a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T20:30:04.284 INFO:journalctl@ceph.mds.k.smithi103.stdout:Apr 26 20:30:04 smithi103 bash[69304]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-k 2024-04-26T20:30:04.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.k.service' 2024-04-26T20:30:04.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:30:04.580 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-26T20:30:04.580 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-26T20:30:04.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.c 2024-04-26T20:30:04.882 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:30:04 smithi204 systemd[1]: Stopping Ceph mds.c for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:30:04.882 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:30:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c[63186]: 2024-04-26T20:30:04.849+0000 7fc84da2a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:30:04.882 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:30:04 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c[63186]: 2024-04-26T20:30:04.849+0000 7fc84da2a700 -1 mds.c *** got signal Terminated *** 2024-04-26T20:30:08.958 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:30:08 smithi204 podman[69230]: 2024-04-26 20:30:08.639438107 +0000 UTC m=+3.874103781 container died 941a731dbf396b39e09599b68541e8b4c1ac282b46b5fbd830f87bfcd061c052 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:30:08.959 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:30:08 smithi204 podman[69230]: 2024-04-26 20:30:08.957368631 +0000 UTC m=+4.192034305 container cleanup 941a731dbf396b39e09599b68541e8b4c1ac282b46b5fbd830f87bfcd061c052 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T20:30:09.257 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:30:08 smithi204 bash[69230]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c 2024-04-26T20:30:09.257 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 26 20:30:09 smithi204 podman[69255]: 2024-04-26 20:30:09.150427092 +0000 UTC m=+0.506332022 container remove 941a731dbf396b39e09599b68541e8b4c1ac282b46b5fbd830f87bfcd061c052 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T20:30:09.272 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.c.service' 2024-04-26T20:30:09.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:30:09.308 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-26T20:30:09.308 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-26T20:30:09.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.f 2024-04-26T20:30:09.620 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:30:09 smithi204 systemd[1]: Stopping Ceph mds.f for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:30:09.881 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:30:09 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f[62880]: 2024-04-26T20:30:09.619+0000 7feb02287700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:30:09.882 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:30:09 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f[62880]: 2024-04-26T20:30:09.619+0000 7feb02287700 -1 mds.f *** got signal Terminated *** 2024-04-26T20:30:12.923 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:30:12 smithi204 podman[69362]: 2024-04-26 20:30:12.644707489 +0000 UTC m=+3.123797240 container died 29b9dc616261e478baf328d4ea332d9ae44cfcb8bf4229b34362ec62cffa0783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T20:30:13.273 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:30:13 smithi204 podman[69362]: 2024-04-26 20:30:13.038433843 +0000 UTC m=+3.517523592 container cleanup 29b9dc616261e478baf328d4ea332d9ae44cfcb8bf4229b34362ec62cffa0783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T20:30:13.273 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 26 20:30:13 smithi204 bash[69362]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-f 2024-04-26T20:30:13.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.f.service' 2024-04-26T20:30:13.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:30:13.449 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-26T20:30:13.449 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-26T20:30:13.449 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.i 2024-04-26T20:30:13.881 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:30:13 smithi204 systemd[1]: Stopping Ceph mds.i for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:30:13.882 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:30:13 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i[63496]: 2024-04-26T20:30:13.758+0000 7f58fb1d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:30:13.882 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:30:13 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i[63496]: 2024-04-26T20:30:13.758+0000 7f58fb1d9700 -1 mds.i *** got signal Terminated *** 2024-04-26T20:30:16.910 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:30:16 smithi204 podman[69486]: 2024-04-26 20:30:16.632523101 +0000 UTC m=+2.965885390 container died ba280396131d3f17c199f7b58db94297df8c1f4ad8814f12d5d17f3f9502980f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5) 2024-04-26T20:30:17.226 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:30:17 smithi204 podman[69486]: 2024-04-26 20:30:17.000323241 +0000 UTC m=+3.333685537 container cleanup ba280396131d3f17c199f7b58db94297df8c1f4ad8814f12d5d17f3f9502980f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T20:30:17.227 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 26 20:30:17 smithi204 bash[69486]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-i 2024-04-26T20:30:17.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.i.service' 2024-04-26T20:30:17.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:30:17.378 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-26T20:30:17.379 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-26T20:30:17.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.l 2024-04-26T20:30:17.631 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:30:17 smithi204 systemd[1]: Stopping Ceph mds.l for 420d0bd6-0407-11ef-bc93-c7b262605968... 2024-04-26T20:30:18.131 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:30:17 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-l[62572]: 2024-04-26T20:30:17.720+0000 7f6cf5e39700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:30:18.132 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:30:17 smithi204 ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-l[62572]: 2024-04-26T20:30:17.720+0000 7f6cf5e39700 -1 mds.l *** got signal Terminated *** 2024-04-26T20:30:20.937 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:30:20 smithi204 podman[69606]: 2024-04-26 20:30:20.935590297 +0000 UTC m=+3.348987063 container died ccaadfeb5f0e07a407008e2941728791dd2f4bf3c09d1be02d3ad8e1c9e56437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T20:30:21.539 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:30:21 smithi204 podman[69606]: 2024-04-26 20:30:21.296112753 +0000 UTC m=+3.709509511 container cleanup ccaadfeb5f0e07a407008e2941728791dd2f4bf3c09d1be02d3ad8e1c9e56437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:30:21.539 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 26 20:30:21 smithi204 bash[69606]: ceph-420d0bd6-0407-11ef-bc93-c7b262605968-mds-l 2024-04-26T20:30:21.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-420d0bd6-0407-11ef-bc93-c7b262605968@mds.l.service' 2024-04-26T20:30:21.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:30:21.696 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-26T20:30:21.696 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T20:30:22.021 INFO:teuthology.orchestra.run.smithi094.stdout:Deleting cluster with fsid: 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:30:27.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T20:30:27.204 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:30:29.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T20:30:29.750 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:30:32.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:30:32.163 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:30:32.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:30:32.221 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T20:30:32.223 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/remote/smithi094/crash 2024-04-26T20:30:32.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/crash -- . 2024-04-26T20:30:32.323 INFO:teuthology.orchestra.run.smithi094.stderr:tar: /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T20:30:32.324 INFO:teuthology.orchestra.run.smithi094.stderr:tar: Error is not recoverable: exiting now 2024-04-26T20:30:32.328 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/remote/smithi103/crash 2024-04-26T20:30:32.329 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/crash -- . 2024-04-26T20:30:32.368 INFO:teuthology.orchestra.run.smithi103.stderr:tar: /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T20:30:32.369 INFO:teuthology.orchestra.run.smithi103.stderr:tar: Error is not recoverable: exiting now 2024-04-26T20:30:32.370 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/remote/smithi204/crash 2024-04-26T20:30:32.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/crash -- . 2024-04-26T20:30:32.404 INFO:teuthology.orchestra.run.smithi204.stderr:tar: /var/lib/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T20:30:32.404 INFO:teuthology.orchestra.run.smithi204.stderr:tar: Error is not recoverable: exiting now 2024-04-26T20:30:32.405 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T20:30:32.406 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-26T20:30:32.512 INFO:tasks.cephadm:Compressing logs... 2024-04-26T20:30:32.513 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T20:30:32.517 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T20:30:32.520 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T20:30:32.543 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T20:30:32.544 INFO:teuthology.orchestra.run.smithi204.stderr:: No such file or directory 2024-04-26T20:30:32.544 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log 2024-04-26T20:30:32.544 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T20:30:32.544 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-26T20:30:32.545 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T20:30:32.552 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T20:30:32.553 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-26T20:30:32.553 INFO:teuthology.orchestra.run.smithi204.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T20:30:32.553 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T20:30:32.553 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-26T20:30:32.553 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-26T20:30:32.553 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mgr.z.log: 89.1% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T20:30:32.553 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T20:30:32.553 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.4% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T20:30:32.555 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T20:30:32.555 INFO:teuthology.orchestra.run.smithi103.stderr:: No such file or directory 2024-04-26T20:30:32.555 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log 2024-04-26T20:30:32.555 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T20:30:32.555 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T20:30:32.556 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-26T20:30:32.556 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose --/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T20:30:32.556 INFO:teuthology.orchestra.run.smithi103.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T20:30:32.556 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T20:30:32.556 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T20:30:32.556 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-26T20:30:32.556 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-26T20:30:32.556 INFO:teuthology.orchestra.run.smithi103.stderr: 89.1% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T20:30:32.557 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.4% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T20:30:32.557 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T20:30:32.557 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-26T20:30:32.557 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.7% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T20:30:32.557 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T20:30:32.561 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.e.log: 93.2% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-26T20:30:32.561 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T20:30:32.562 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-volume.log: 94.4% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-26T20:30:32.562 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-26T20:30:32.562 INFO:teuthology.orchestra.run.smithi204.stderr: 93.3% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-26T20:30:32.562 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-26T20:30:32.563 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-26T20:30:32.563 INFO:teuthology.orchestra.run.smithi204.stderr: 82.7% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T20:30:32.563 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T20:30:32.569 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-26T20:30:32.571 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.k.log: 91.0% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T20:30:32.572 INFO:teuthology.orchestra.run.smithi094.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-26T20:30:32.572 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T20:30:32.573 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-26T20:30:32.574 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log 2024-04-26T20:30:32.575 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T20:30:32.577 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-26T20:30:32.577 INFO:teuthology.orchestra.run.smithi094.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-26T20:30:32.578 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-26T20:30:32.578 INFO:teuthology.orchestra.run.smithi103.stderr: 94.4% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-26T20:30:32.579 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T20:30:32.580 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.g.log: /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.j.log: 89.0% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T20:30:32.581 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T20:30:32.581 INFO:teuthology.orchestra.run.smithi094.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T20:30:32.582 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T20:30:32.582 INFO:teuthology.orchestra.run.smithi094.stderr: 90.2% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T20:30:32.583 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T20:30:32.584 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T20:30:32.586 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mon.a.log: 94.4%/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.6.log: -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-26T20:30:32.587 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T20:30:32.587 INFO:teuthology.orchestra.run.smithi094.stderr: 94.4% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-26T20:30:32.589 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-26T20:30:32.592 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T20:30:32.592 INFO:teuthology.orchestra.run.smithi094.stderr: 82.8%/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.a.log: -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T20:30:32.594 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.l.log: 94.4% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-26T20:30:32.595 INFO:teuthology.orchestra.run.smithi204.stderr: 91.0% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T20:30:32.597 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-26T20:30:32.599 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-26T20:30:32.605 INFO:teuthology.orchestra.run.smithi204.stderr: 94.2% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-26T20:30:32.614 INFO:teuthology.orchestra.run.smithi204.stderr: 94.6% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-26T20:30:32.615 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.d.log: /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.9.log: 90.9% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T20:30:32.616 INFO:teuthology.orchestra.run.smithi094.stderr: 94.5% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-26T20:30:32.618 INFO:teuthology.orchestra.run.smithi204.stderr: 94.6% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-26T20:30:32.638 INFO:teuthology.orchestra.run.smithi103.stderr: 94.5% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-26T20:30:32.670 INFO:teuthology.orchestra.run.smithi094.stderr: 93.9% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-26T20:30:32.755 INFO:teuthology.orchestra.run.smithi103.stderr: 93.8% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-26T20:30:32.943 INFO:teuthology.orchestra.run.smithi204.stderr: 92.0% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T20:30:32.975 INFO:teuthology.orchestra.run.smithi094.stderr: 90.5% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T20:30:33.113 INFO:teuthology.orchestra.run.smithi103.stderr: 91.9% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T20:30:33.497 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T20:30:33.502 INFO:teuthology.orchestra.run.smithi204.stderr: 92.7% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T20:30:33.569 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T20:30:33.576 INFO:teuthology.orchestra.run.smithi103.stderr: 92.3% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-26T20:30:33.678 INFO:teuthology.orchestra.run.smithi094.stderr: 92.8% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T20:30:33.682 INFO:teuthology.orchestra.run.smithi204.stderr: 93.7% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-26T20:30:33.788 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T20:30:33.880 INFO:teuthology.orchestra.run.smithi204.stderr: 93.4% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-26T20:30:33.882 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-26T20:30:33.882 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m1.348s 2024-04-26T20:30:33.882 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m4.878s 2024-04-26T20:30:33.882 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m0.187s 2024-04-26T20:30:33.990 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T20:30:33.992 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-26T20:30:33.992 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m1.457s 2024-04-26T20:30:33.992 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m5.366s 2024-04-26T20:30:33.992 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.208s 2024-04-26T20:30:34.043 INFO:teuthology.orchestra.run.smithi094.stderr: 90.2% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T20:30:34.073 INFO:teuthology.orchestra.run.smithi094.stderr: 93.4% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-26T20:30:34.193 INFO:teuthology.orchestra.run.smithi094.stderr: 93.0% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T20:30:34.294 INFO:teuthology.orchestra.run.smithi094.stderr: 93.4% -- replaced with /var/log/ceph/420d0bd6-0407-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T20:30:34.298 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-04-26T20:30:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:real 0m1.749s 2024-04-26T20:30:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:user 0m7.220s 2024-04-26T20:30:34.299 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m0.692s 2024-04-26T20:30:34.300 INFO:tasks.cephadm:Archiving logs... 2024-04-26T20:30:34.300 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/remote/smithi094/log 2024-04-26T20:30:34.301 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T20:30:35.304 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/remote/smithi103/log 2024-04-26T20:30:35.305 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T20:30:36.124 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/remote/smithi204/log 2024-04-26T20:30:36.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T20:30:36.894 INFO:tasks.cephadm:Removing cluster... 2024-04-26T20:30:36.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 --force 2024-04-26T20:30:37.195 INFO:teuthology.orchestra.run.smithi094.stdout:Deleting cluster with fsid: 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:30:37.755 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 --force 2024-04-26T20:30:37.947 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:30:38.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 420d0bd6-0407-11ef-bc93-c7b262605968 --force 2024-04-26T20:30:38.390 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 420d0bd6-0407-11ef-bc93-c7b262605968 2024-04-26T20:30:38.660 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T20:30:38.661 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T20:30:38.716 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T20:30:38.731 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T20:30:38.747 INFO:tasks.cephadm:Teardown complete 2024-04-26T20:30:38.748 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T20:30:38.757 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T20:30:38.758 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T20:30:38.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T20:30:38.774 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T20:30:38.944 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T20:30:38.953 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T20:30:38.954 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T20:30:38.962 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T20:30:38.965 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T20:30:38.976 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-26T20:30:38.978 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-04-26T20:30:38.981 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T20:30:38.981 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-26T20:30:38.982 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv01.front.sepia.ceph.com 3 6 377 51 +1389us[+1387us] +/- 57ms 2024-04-26T20:30:38.982 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 51 -1014us[-1014us] +/- 75ms 2024-04-26T20:30:38.982 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -953us[ -953us] +/- 87ms 2024-04-26T20:30:38.982 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:30:38.983 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T20:30:38.983 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-04-26T20:30:38.984 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv01.front.sepia.ceph.com 3 7 377 116 +1351us[+1356us] +/- 56ms 2024-04-26T20:30:38.984 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -1083us[-1083us] +/- 75ms 2024-04-26T20:30:38.984 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 -1015us[-1015us] +/- 86ms 2024-04-26T20:30:38.984 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:30:39.008 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-04-26T20:30:39.019 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T20:30:39.019 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-04-26T20:30:39.019 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv01.front.sepia.ceph.com 3 8 377 251 +1303us[+1307us] +/- 54ms 2024-04-26T20:30:39.020 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 55 -1145us[-1145us] +/- 75ms 2024-04-26T20:30:39.020 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -1087us[-1087us] +/- 87ms 2024-04-26T20:30:39.020 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:30:39.021 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T20:30:39.033 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T20:30:39.034 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T20:30:39.052 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T20:30:39.261 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T20:30:39.324 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T20:30:39.388 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T20:30:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:30:39.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T20:30:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:30:39.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T20:30:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:30:39.538 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T20:30:39.548 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T20:30:39.578 INFO:teuthology.task.internal:Duration was 2433.740316 seconds 2024-04-26T20:30:39.579 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T20:30:39.592 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T20:30:39.593 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T20:30:39.601 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T20:30:39.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T20:30:39.637 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T20:30:39.639 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T20:30:39.741 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T20:30:40.332 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T20:30:40.332 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-04-26T20:30:40.332 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T20:30:40.396 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T20:30:40.396 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T20:30:40.426 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-04-26T20:30:40.427 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T20:30:40.449 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T20:30:40.449 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T20:30:40.456 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T20:30:40.469 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T20:30:40.539 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T20:30:40.540 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T20:30:40.545 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T20:30:40.547 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T20:30:41.031 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T20:30:41.041 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T20:30:41.042 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T20:30:41.123 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T20:30:41.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T20:30:41.173 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T20:30:41.183 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:30:41.191 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:30:41.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:30:41.213 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-04-26T20:30:41.235 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-04-26T20:30:41.284 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-04-26T20:30:41.326 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:30:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:30:41.374 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:30:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:30:41.389 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:30:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:30:41.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T20:30:41.413 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T20:30:41.414 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/remote/smithi094 2024-04-26T20:30:41.414 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T20:30:41.521 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/remote/smithi103 2024-04-26T20:30:41.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T20:30:41.556 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112/remote/smithi204 2024-04-26T20:30:41.556 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T20:30:41.592 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T20:30:41.592 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T20:30:41.600 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T20:30:41.603 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T20:30:41.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T20:30:41.661 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T20:30:41.661 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T20:30:41.706 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T20:30:41.706 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T20:30:41.714 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T20:30:41.716 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T20:30:41.728 INFO:teuthology.orchestra.run.smithi103.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 20:30 /home/ubuntu/cephtest 2024-04-26T20:30:41.730 INFO:teuthology.orchestra.run.smithi204.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 20:30 /home/ubuntu/cephtest 2024-04-26T20:30:41.762 INFO:teuthology.orchestra.run.smithi094.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 20:30 /home/ubuntu/cephtest 2024-04-26T20:30:41.766 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T20:30:41.778 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T20:30:41.812 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T20:30:41.839 DEBUG:teuthology.nuke:shortname: smithi094 2024-04-26T20:30:41.840 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T20:30:41.864 DEBUG:teuthology.nuke:shortname: smithi103 2024-04-26T20:30:41.864 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T20:30:41.869 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 19:36:39.723675', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNZ74+Dji4nEdGTf+G1+r1tjbiHk5k11agowi/sxtvG2lee+FcSSeyKxEZzqAqMYFZWvUOCpbJ88fpB/Goplncc='} 2024-04-26T20:30:41.888 DEBUG:teuthology.nuke:shortname: smithi204 2024-04-26T20:30:41.889 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T20:30:41.894 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 19:36:39.725692', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1ZHiyP88AahpnNFyE3urr2om9yXoPF99DDZUeHQuendNaWgnhW8SihPzfD76Yu9oH+rTo6biiOMauadRw2ID4='} 2024-04-26T20:30:41.923 INFO:teuthology.orchestra.console.smithi094:Power off 2024-04-26T20:30:41.923 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T20:30:41.938 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675112', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 19:36:39.724790', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJ4ay40E+CaHlLU+1/uxRQ/X7vqdPtpZN8K7JPya5CRaFk37mKhzqngXSxm4JZbsiU1sbOAzRaYkLobwTzTun0='} 2024-04-26T20:30:41.950 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2024-04-26T20:30:41.950 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:41.970 INFO:teuthology.orchestra.console.smithi103:Power off 2024-04-26T20:30:41.970 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T20:30:41.980 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-04-26T20:30:41.993 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-04-26T20:30:41.994 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:42.021 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-26T20:30:42.033 INFO:teuthology.orchestra.console.smithi204:Power off 2024-04-26T20:30:42.034 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T20:30:42.058 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-04-26T20:30:42.058 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:42.082 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-04-26T20:30:45.981 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:46.023 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:46.084 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:46.094 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-04-26T20:30:46.134 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-04-26T20:30:46.195 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-04-26T20:30:50.098 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:50.134 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:50.196 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:50.209 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-04-26T20:30:50.244 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-04-26T20:30:50.306 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-04-26T20:30:50.345 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-04-26T20:30:50.407 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-04-26T20:30:54.214 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:30:54.325 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2024-04-26T20:30:54.426 INFO:teuthology.orchestra.console.smithi094:Power off completed 2024-04-26T20:30:54.531 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} duration: 2433.7403161525726 failure_reason: 'Command failed on smithi094 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c06fdcd85ecc4404925ad110487ee4a0 status: fail success: false 2024-04-26T20:30:54.532 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T20:30:54.613 INFO:teuthology.run:FAIL